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ABSTRACT 

Recent trends indicate a significant decline in the number of students graduating 
from Science, Technology, Engineering and Math (STEM) programs in the US. The 
under-representation of students of color, females and low income students in STEM 
programs has also been documented. Design Based Science (DBS) instruction has been 
suggested to improve the problem solving skills of students of color. 

The present study employed a quasi-experimental pre-post-test research study. 
Four equivalent parallel high school traditional Chemistry classes of eighty two (82) 10 l 
and 1 1 th grade students was invited to participate in this study. The treatment group 
comprised of 36 students while the control group was made up of 46 students. 

The purpose of this study was to investigate whether DBS affects student problem 
solving competency and chemistry achievement across student demographics (gender, 
race and SES). The research questions were: 1) Does DBS have any effect on the 
problem solving competencies of students in a high school traditional chemistry class? 2) 
Does the effect of DBS on problem solving competency depend on gender? 3) Does the 
effect of DBS on problem solving competency depend on race? 4) Does the effect of 
DBS on problem solving competency depend on SES? 5) Does DBS have any effect on 
the chemistry achievement of students in a high school traditional chemistry class? 6) 
Does the effect of DBS on chemistry achievement vary depending on gender? 7) Does 



the effect of DBS on chemistry achievement vary depending on race? 8) Does the effect 
of DBS on chemistry achievement vary depending on SES? 9) Is the problem solving 
competency of students in a traditional chemistry class predictive of their chemistry 
achievement? 

The findings are as follow: a) DBS significantly improved the problem solving 
competency of students in the study, b) DBS significantly improves the problem solving 
competency of both males and females, with a slight urge among females, c) the 
differences in the effects of DBS in improving problem solving competency among Black 
and Hispanic students in this study was not statistically significant, however, Black 
students and Hispanic female students showed significant improvement in problem 
solving competency after the DBS instruction, d) DBS did not statistically significantly 
improve the problem solving competency of students of particularly SES group(s), and e) 
Problem solving competency is a strong predictor of higher chemistry concepts score 
among students in both treatment and control groups. 
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CHAPTER I 
INTRODUCTION 

The total number of students graduating from math, engineering and physical 
science majors has been on the decline since the mid-1980s (Mooney and Laubach, 
2002). Recent trends suggest a significant decline in the number of students interested in 
Science, Technology, Engineering and Mathematics (STEM) careers (van Langen and 
Dekkers, 2005) in K-12. The under representation of STEM Bachelor degrees earned by 
targeted minority student groups, namely, African Americans, Latino/as, South East 
Asians and Native Americans (ALANAS) (The Center for Education and Work, 2008) 
and girls (Fadigan & Hammrich, 2004; Gilbert & Calvert, 2003; Scantlebury & Baker, 
2007) has also been reported. Despite the similarity in the intentions of ALANA and 
White students to major in STEM fields the former are less likely to major and more 
likely to drop out of STEM programs (The Center for Education and Work, 2008). A 
number of factors have been identified as accounting for these findings. 

One of the reasons advance for the decline in the number of US students 
participating in STEM programs, in general and in particular among female and minority 
students is the view among these students that science and technology is uninteresting, 
difficult and closed off to them (President's Council of Advisors on Science and 
Technology - PCAST, 2010). Hanushek & Rivkin, (2003) also point to teacher quality 
and effectiveness as important factors determining students' performance in STEM 
classrooms. The results of the PISA 2003 problem solving competency assessment 
(OECD, 2003) suggest a strong correlation between problem solving competency and 
socioeconomic status (SES). With a high proportion of ALANAS falling within low SES 



in the United States, the PISA 2003 problem solving assessment results also suggest low 
problem solving competency among this group of students. The current study focuses on 
problem solving ability since problem solving has been identified as a 21 st century skill 
needed for students to be successful in and out of school (Partnership for 21 st Century 
Skills, 2009). Problem solving is a focus of the current study also because problem 
solving through scientific reasoning (as proposed by John Dewey) remains a primary goal 
of science education (Atkins & Black, 2003). 

Various related pedagogies such as Problem-based learning (P1BL), Project-based 
learning (PjBL), Design-Based Science (DBS) and Inquiry-based learning (IBL) have 
been used in an attempt to improve problem solving skills. These pedagogies 
contextualize learning by making content relevant to student life experiences. Such 
modes of teaching have a potential of closing the achievement gap (Wisely, 2009). 
During Problem-Based Learning students are presented with an "ill- structured" problem 
intended to help students build skills and content knowledge. PjBL is similar to P1BL 
except that student projects involve a culminating artifact in the problem solving lesson. 
DBS is a type of PjBL, which incorporates inquiry (the use of the scientific method in 
problem solving). 

In his review of research on PjBL, Thomas (2000) notes that most of the research 
on PjBL took place between 1995 and 2000. Thomas (2000) references a number of 
studies such as those by the Expeditionary Learning Outward Bound (ELOB), Co-nect 
schools and the Academy for Educational Development (AED), relating PjBL and 
student achievement. Although significant improvements in student achievement were 
reported by the studies for students in PjBL programs, Thomas (2002) points out that the 



results may be attributable in part to features other than those of PjBL (e.g. portfolios, 
flexible block scheduling for ELOB; technology in the case of Co-nect schools). He also 
posits that since technology and expeditions do not target basic skills (reading, writing 
and computation), the reported effects of these PjBL-programs on student's basic skills 
achievement may be the result of a generalized effect associated with the whole school 
reform effort or, perhaps, the motivational effect of project-based instruction may lead to 
increased student attendance, attention, and engagement during the (non-project) periods 
students spend learning basic skills. Also, only one study of PjBL effectiveness was 
found that used a longitudinal experimental design, with pre-post assessments. As far as 
the effects of PjBL on problem solving skills, studies conducted between 1995 and 2000 
relate primarily to P1BL (not PjBL) (Thomas, 2000). Furthermore, there are improved 
methods for assessing problem solving capabilities (OECD, 2003). Recent empirical 
studies suggest that middle school students, particularly African-American students, 
involved in DBS improve both in their science content knowledge and problem solving 
skills (Fortus, Dershimer, Krajcik, Marx and Mamlok-Naaman, 2005; Mehalik, Doppelt 
and Schuun , 2008). It is clear from the preceding overview that while more research is 
needed in the effectiveness of PjBL (and consequently DBS) in improving student 
achievement and problem solving capacities, more of this work is needed at the high 
school level, where achievement gaps, particularly that between genders widen (Ingels & 
Dalton, 2008; Bacharach, Baumeister & Furr, 2003; Jones, Mullis, Raizen, Weiss, 
& Weston, 1992). Also, in the last ten years, just a few studies have been conducted 
comparing the effectiveness of DBS in improving science achievement across 
race/ethnicity. The focus of this study is thus to investigate the relationships between 



DBS on the one hand, and problem solving competency and science knowledge gain on 
the other, among different genders, race and socioeconomic groups. 

The interactions between DBS problem solving skills, science achievement and 
race-ethnicity, elicited from some of the above referenced studies, are summarized in 
Figure I below. Mehalik, Doppelt and Schunn 2008; Fortus, Dershimer, Krajcik, Marx 
and Mamlok-Naaman, 2004, 2005 and Chang, 2001a, 2001b) posit that different modes 
of problem- solving associated instruction such as DBS can improve students' problem- 
solving ability and consequently science achievement. The strong correlation between 
problem solving ability and science achievement reported by OECD (2003) also speaks 
to the latter relationship. DBS is one of such science pedagogies that has been found to 
show promising results in science education, by improving both science achievement 
(Kolodner, Camp, Crismond, Fasse, Gray, Holbrool, Puntambekar, Ryan, 2003; Mehalik, 
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Figure I. Interactions between DBS, Science Knowledge and problem-solving skills 



Doppelt and Schunn, 2008; Fortus, Dershimer, Krajcik, Marx and Mamlok-Naaman, 
2004, 2005; Silk, Schunn & Cary (2009). DBS has also been described as pedagogy in 
which scientific knowledge and problem-solving skills are constructed. 

The extent to which the achievement gap contributes to the potential loss of the 
United State's global competitive edge as well as maintenance of a robust national 
economy cannot be overlooked. The McKinsey consulting firm released a report on the 
economic impact of the achievement gap in America's Schools in 2009. The implications 
of the report revealed a national decline in productivity and jobs. According to the report, 
if the U.S. had been able to close the gap in science and math achievement between 1983 
and 1998 and raised its performance to the level of such nations as Canada, Finland and 
South Korea, the U.S. Gross Domestic Product (GDP) in 1998 would have been 
approximately $2 trillion higher. If the achievement gap had been closed between Black 
and Hispanic students on the one hand and white and Asian students by 1998, the GDP in 
2008 would have been about $400 to $500 billion higher. If the gap between America's 
low-income students and the remaining students had been similarly narrowed, GDP in 
2008 would have been $400 to $670 billion higher. In terms of PISA math and science 
output and the amount of money the U.S. spends on each student, which is among the 
highest in the world, the report concludes that the United States gets 60% less for its 
education dollars in terms of average test score results than do other wealthy 
[industrialized] nations. 

This chapter presents the trend of declining science achievement among U.S. 
students as well as the potential of DBS in the construction of new scientific knowledge. 
It also reviews the potential of DBS in the development of problem solving skills in 



secondary science education. It describes the need for a study of the effects of DBS on 
different groups of students, vis-a-vis the over-a-decade long decline in the number of US 
students in STEM programs (particularly girls, and minority students) as well as poor 
problem solving abilities. This decline is presented not only as a national issue but also in 
relation to other industrialized countries with its implications for the U.S. economy. A 
number of contributory factors to the decline are discussed, with emphasis on the 
problem- solving abilities of students across gender, racial-ethnic and socioeconomic 
groups both in the U.S. and on the international scene. Also, findings from empirical 
studies are presented in exploring the benefits of innovative science pedagogies on the 
development of problem- solving skills. The relationships between such pedagogies as 
DBS, P1BL, project-based learning, and inquiry that highlight problem-solving and the 
development of problem- solving skills are presented. The goal of this chapter is therefore 
to propose the need to study the effects of DBS (an integration of project-based learning 
and inquiry-based learning) on high-school students' problem solving skills and science 
knowledge. Problem- solving skills must be studied across gender, racial-ethnic and SES 
in an attempt to meaningfully contribute to efforts aimed at closing the achievement gap 
in science. Closing the achievement gap has become increasingly important in a 
struggling U.S. education system in which racial-ethnic and socio-economic differences 
are becoming more and more pronounced. 
Trends in the Performance of U.S. Students in Science 

The decline in U.S. education has been attributed to inadequate preparation and 
experiences in math and sciences (ACT, 2006), family characteristics and educational 
support variables, attitudes toward math and science, differences in aptitude (Benbow & 



Arjmand, 1990) and inadequate problem-solving skills (Ornstein, 2010). Although the 
United States can identify individual schools and school districts that have been 
successful, the U.S. education system as a whole has been on the decline for at least a 
couple of decades nationally and internationally. The Department of Education (2004) 
and Helpman (2004) reported that several indicators of the performance of U.S. students 
in science and mathematics education at the pre-college level reveal a mixed picture of 
successes and shortcomings. A discussion of data from the National Assessment of 
Educational Progress (NAEP), the Trends in International Mathematics and Science 
Study (TIMSS) and the Program for International Student Assessment (PISA) will 
elucidate the performance of students in high stakes science assessments. 

On NAEP, less than one-third of U.S. eighth graders show proficiency in 
mathematics and science, and science test scores have improved although very little over 
the past few decades. According to the US Department of Education (2012), the overall 
average score for the nation at grade 8 was 2 points higher in 201 1 than in 2009. Score 
gaps between White and Black students and White and Hispanic students narrowed from 
2009 to 201 1. Sixty-five percent of eighth-graders performed at or above the in 201 1, 32 
percent performed at or above Proficient, and 2 percent performed at the Advanced level. 
The percentages of students at or above Basic and at or above Proficient were higher in 
201 1 than in 2009. Of the 47 states/jurisdictions that participated in 2009 and 201 1, 
public-school students in 16 states scored higher in 2011 than in 2009. In 2011, students 
in 29 states scored higher than the national average, and in 16 states they scored lower. 
The NAEP 2005 science assessments revealed that 12 l graders, showed no change in 
performance from the administration of the assessment in 2000. However, the 2005 



average scores were lower than those in 1996. Also, at this grade level, the percentage of 
students performing at or above the basic level, at or above the proficient level, and at the 
advanced level all declined compared to 1996 data. In addition, the number of students 
who scored below basic increased since 1996 (Department of Education, 2006). In the 
U.S., achievement gaps in science and math between white or Asian/Pacific Islander 
students and minorities (traditionally underrepresented in STEM) exist at all levels, 
including significant gaps among the highest-performing students. For example, a recent 
analysis of both NAEP and state assessment data shows that large achievement gaps in 
mathematics performance continue to persist between white and underrepresented 
minority high achievers (Plucker, Burroughs & Song, 2010). 

International comparisons of our students' performance in science and 
mathematics place the United States in the middle of the pack or lower. TIMSS measures 
what students know and can remember in science and math. In 2007 U.S. fourth graders 
and eighth graders placed about average among industrialized and rapidly industrializing 
countries. However, U.S. students in fourth, eighth, and twelfth grades drop progressively 
lower on international comparisons of science and mathematics ability as their grade 
level increases. The U.S. Department of Education (2009), reports that in 2007, the 
average science scores of both U.S. fourth-graders (539) and eighth-graders (520) were 
higher than the TIMSS scale average (500 at both grades). The average U.S. fourth-grade 
science score was higher than those of students in 25 of the 35 other countries, lower than 
those in 4 countries (all of them in Asia), and not measurably different from those in the 
remaining 6 countries. At eighth grade, the average U.S. science score was higher than 
the average scores of students in 35 of the 47 other countries, lower than those in 9 



countries (all located in Asia or Europe), and not measurably different from those in the 
other 3 countries. 

PISA is a triennial survey of knowledge and skills of fifteen- year old students in 
countries that form the Organization for Economic Co-operation and Development 
(OECD). The areas PISA focuses on during each six-year cycle are mathematics, science, 
reading and problem solving. However, within each cycle the surveys emphasize 
different areas. The U.S. average score in science literacy in 2009 was higher than the 
U.S. average in 2006, the only time point to which PISA 2009 performance can be 
compared in science literacy. While U.S. students scored lower than the OECD average 
in science literacy in 2006, the average score of U.S. students in 2009 was not 
measurably different from the 2009 OECD average. However, in 2009 while 1% of U.S. 
students were proficient at level 6 the percentage was higher (1.5%) in 2006. At Level 6, 
students can consistently identify, explain and apply scientific knowledge and knowledge 
about science in a variety of complex life situations. U.S. students scored below most 
other nations tested in 2006, and the U.S. standing dropped from 2000 to 2006 in both 
math and science. 

It is generally accepted that most high school graduates do not enroll in science 
and mathematics courses. In fact, of all ninth graders in the United States in 2001, for 
example, only about 4 percent are predicted to earn college degrees in STEM fields by 
201 1 (PC AST, 2010). Likewise, many studies note the importance of achievement in 
science during high school for determining later persistence in the science pipeline 
through college and early career years (Carmichael, 2007; Hanson, 1996; Kaufman, 
1991). Other predictors of persistence in the science pipeline include classroom climate, 



classroom pedagogy, faculty attitudes and behavior and financial aid (Carmichael, 2007). 
Thus, pre-college years represent a critical period for encouraging students to enter the 
science pipeline. Indeed, the level of science preparation in secondary school, and 
specifically pre-college science achievement, is generally noted to be among the most 
consistent and best predictors of students' interest and persistence in postsecondary 
STEM fields (Griffith, 2010; Hanson, 1996; Davis, Ginorio, Hollenshead, Lazarus & 
Rayman., 1996). 

The decline in the stream of high school graduates into undergraduate science, 
technology, engineering and mathematics trend manifests on college campuses all over 
the country. In fact, the number of U.S. college students majoring in science, math and 
engineering is flat, and the percentage of graduates in these essential areas in Western 
European and especially Asian countries have increasingly outpaced the U.S. (Ornstein 
2010). The achievement gaps in science and math between white or Asian/Pacific 
Islander students and minorities as established by NAEP and state assessment data 
underscore a systemic problem: the lack of opportunities and support for 
underrepresented minority students, inadequate teaching, and an absence of both real-life, 
hands-on experiences with STEM materials and positive role models of STEM 
professionals (Hanushek & Rivkin, 2009; Reardon, 2008; Gandara, 2005; Donovan & 
Cross, 2002). In addition, many students who academically qualify for postsecondary 
studies in science and math fields at both two- and four-year institutions don't pursue 
those programs due to a number of reasons. These reasons include: dissuasion by 
disappointing postsecondary experiences, high tuition or demanding curricula and 
courses of study, relatively low salaries in STEM fields compared to other professions, or 
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the lack of role models with whom they can identify (American Association of State 
Colleges and Universities, 2005). 

As varied as the causes are for a struggling U.S. education system, so must the 
solution. The under-representation of women, people of color, and the poor decreases not 
just the quantity, but the quality and breadth of the talent of persons in STEM fields 
(Drew, 1996). Just by their sheer numbers, women, people of color and the poor 
participate significantly in the struggling U.S. education system. Solutions for the 
declined U.S. education system must address the achievement gap in science and math 
between races, socio-economic classes and gender. The achievement gap between Asian 
and white students compared to Hispanic and black students remain alarmingly high, 
granted that by the year 2015 the latter group of students will represent the majority 
enrollments in U.S. public schools (Ornstein 2010). It is typically noted in the literature, 
the primary cause of attrition of minority students from scientific fields is often poor 
academic preparation prior to college (Oakes, 1990; Petersdorf, 1991). In addition, 
racial-ethnic differences in science achievement are generally larger throughout all 
grades than are gender differences (Hanson, 1996). Research studies show that race- 
ethnicity explains much more of the variance in science achievement scores than does 
gender, and females and males within racial-ethnic categories are much more similar 
with regard to achievement than are females across racial-ethnic categories (Muller, 
Stage & Kinzie, 2001; Clewell & Ginorio, 1996; Creswell & Houston, 1980). Research 
on racial-ethnic differences shows that Asian American and White students show higher 
science achievement scores, as well as disproportionately greater science achievement 
gains, during middle school and high school than their Latino/a and African American 
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counterparts (Bacharach, Baumeister & Furr, 2003; Scott, Rock, Pollack, Ingels, & 
Quinn, 1995), and are similarly overrepresented in high school and college science 
courses; STEM college majors; and scientific and technical careers (Peng, Wright, & 
Hill, 1995). In general, these racial-ethnic differences on standardized science tests 
appear much earlier than gender differences (Bacharach, Baumeister & Furr, 2003; 
Mullis, Dossey, Owen, & Phillips, 1993), and the racial-ethnic differences tend to 
increase with age (Bacharach, Baumeister & Furr, 2003; Gross, 1988, 1989). Although 
less prominent, gender gaps in science must be addressed to ensure that all students reach 
their potential. 
Gender Gap in STEM Education 

Over the last 30 years the gender gap in science has narrowed, however, girls and 
women remain underrepresented and marginalized in physics, engineering and 
technology (Fadigan & Hammrich, 2004; Gilbert & Calvert, 2003; Scantlebury & Baker, 
2007). The gender gap favoring males does not only appear consistently across all racial- 
ethnic groups, but also the pattern appears to be consistent throughout middle school and 
high school, with the differences widening sharply by Grade 12 (Ingels & Dalton, 2008; 
Bacharach, Baumeister & Furr, 2003; Jones, Mullis, Raizen, Weiss, &Weston, 1992). 
There are a number of pointers that may explain some of these observations. Girls of all 
ethnic groups have negative science attitudes and fewer science experiences than boys 
(Catsambis, 1995). According to Miller, Blessing & Schwartz (2006) girls like biology, 
chose people-oriented majors, and chose science majors to help people or animals and 
that girls perceive science as uninteresting, passionless, or leading to an unattractive 
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lifestyle. This finding partly explains the under-representation of women in STEM 
professions. 

In 2006, women earned only 28% of Ph.D. s in physical sciences, 25% in 
mathematics and computer science, and 20% in engineering in the United States (NSF, 
2008). Although women made up 47% of the US workforce in 2009, the percentage of 
women in lucrative technical professions, such as "computer and mathematical 
occupations" and "architecture and engineering occupations," reached only 25% and 
14%, respectively (US Bureau of Labor Statistics, 2009). The gender gap in STEM 
disciplines goes beyond the limited representation of women. In college physics women 
earn lower exam grades and lower scores on standardized tests of conceptual mastery 
(Pollock, Finkelstein & Kost, 2007; Brewe, Sawtelle, Kramer, O'Brien, Rodriguez, & 
Pamela, 2010). 

Various factors have been identified as accounting for the gender gap in STEM. 
Students' prior background and preparation in mathematics and physics have been 
identified as a major contributor to performance in introductory physics (Hazari, Tai & 
Sadler, 2007). In their review of literature Scantlebury and Baker (2007) identified other 
factors such as science attitudes, classroom environments and the impact of policies such 
as high-stakes testing that contribute to the gender gap. Brotman & Moore (2008), in 
reviewing science education literature from 1995 to 2006, developed four themes 
underlying the gender gap a) equity and access b) curriculum and pedagogy c) the nature 
and culture of science and d) identity. 

Strategies related to gender responsive curricula are found in PjBL (and DBS) and 
is therefore worth exploring. Brotman & Moore (2008) noted a variation in what different 
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researchers described as gender-inclusive curriculum and pedagogy. However, some 
common features emerged from the interventions attempted by these researchers. 
Specifically, a gender-inclusive science curriculum draws upon girls' and boys' 
experiences, interests, and preconceptions; prioritizes active participation; incorporates 
long-term, self-directed projects; includes open-ended assessments that take on diverse 
forms; emphasizes collaboration and communication; provides a supportive environment; 
uses real-life contexts; and addresses the social and societal relevance of science. It also 
pays attention to issues of sexism and gender bias in curriculum materials. Roychoudhur, 
Tippins, & Nichols (1995) found that among majority of prospective elementary teachers 
- most of whom were female, situated, collaborative learning and long-term, open-ended 
projects in a physical science class triggered feelings of empowerment, competence, 
ownership, and an appreciation for the connection between science and their lives. The 
implications of the achievement gap to the United States make it imperative for serious 
systemic corrective actions. Gender- inclusive science curricula are in many ways 
consistent with recommendations made by science education reform efforts in general, 
which attempt to improve science education for all students through constructivist 
approaches (American Association for the Advancement of Science [AAAS, 1993]; 
National Research Council [NRC, 1996]) promoted by John Dewey, for example, DBS. 
John Dewey and Problem Solving 

In 1910 Dewey proposed problem-solving through scientific reasoning as a goal 
of science education. A review of science education reform in the U.S. and other 
developed nations (Atkin and Black 2003) clearly indicate that this goal was never really 
abandoned. Dewey's goal of science education was punctuated in the 1950s by the goal 
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of teaching science for science's sake, led by the University of Illinois. However, by the 
late 1980s the Deweyan problem-solving approach to science education, revived in the 
U.S., was evident in other developed nations such as France, Japan, Scotland, Canada, 
Australia, Germany and Spain. As the 20 £ century drew to a close the Deweyan goal of 
science education was broadened to include inquiry. During the last fifteen years inquiry 
activities have become increasingly integrated with the design process, which is also a 
problem- solving process (Fortus, Dershimer, Krajcik, Marx & Mamlok-Naaman (2004). 
This is in line with recommendations by the International Technology Education 
Association (ITEA) (ITEA, 2002). 

The development of problem-solving skills as a long-standing goal of science 
education is well documented (Atkin and Black 2003, Stewart, 1982; Wavering, 1980; 
Champagne and Klopfer, 1977). In furtherance of this goal however, school science has 
traditionally been taught around well-defined problems, such as predicting an ideal 
projectile's trajectory or calculating how much water is generated by the ignition of given 
amounts of hydrogen and oxygen. On the other hand, real- world scientific inquiry 
focuses on ill-defined problems as aptly described by the American Association for the 
Advancement of Science (AAAS, 1990), 

"There simply is no fixed set of steps that scientists always follow, no one path 

that leads them unerringly to scientific knowledge" (p. 4). 
Many school curricula and teaching practices have been criticized because they do not 
give students experience in real- world problems, in situations where decisions are not 
clear-cut, where requirements can conflict, and optimization rather than 'proof is needed. 
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A number of researchers and organizations have recommended restructuring 
school science so that science, in the classroom is taught around real-world problems 
relevant to students' lives. Thus design activities or lessons, that involve real-world 
problem solving, should be incorporated into education in general and in science 
education in particular (AAAS, 1990; Chiapetta, Koballa, Jr., & Collette, 2002; Davis, 
1998; ITEA, 2002; Layton, 1993; NRC, 2002). These recommendations have led to 
crucial actions by various stakeholders including the U.S. federal government, industry 
and foundations to improve science, technology, engineering and mathematics (STEM) 
education. Real-world problems are ill-defined, lacking some required information, and 
not necessarily having a known correct or the best solution (Nickerson, 1994; Roberts, 
1995). 

The Government Accountability Office (GAO, 2005) catalogued and assessed the 
impact of federal programs designed to improve educational programs, particularly 
STEM curricula. The analysis also included the impact of such programs on the number 
of students pursuing STEM careers. Industries and firms dependent upon a strong science 
and math workforce pipeline have launched a variety of programs that target K-12 
students and undergraduate and graduate students in STEM fields. Industry associations 
that include the Society for Manufacturing Engineers, the American Chemical Society, 
the American Physical Society, the National Association of Manufacturers, and the 
National Science and Technology Education Partnership invest in STEM education 
initiatives that involve curricular improvements, career-focused websites, mentoring 
programs, scholarships, and other incentives and supports. Individual firms and their 
corporate foundations, including Raytheon, Bayer, and General Electric, have created 
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outreach efforts of their own (Delaware Valley Industrial Resource Center and National 
Council for Advanced Manufacturing, 2006). Project Lead the Way (PLTW) operates in 
over 4,000 middle and high schools in the 201 1/12 school year in all 50 states of the U.S., 
bringing them STEM programs (PLTW, 201 1). For instance, The PLTW Gateway To 
Technology (GTT) program features a project-based curriculum designed to challenge 
and engage the natural curiosity and imagination of middle school students 
( http://www.pltw.org/our-programs/middle-school-engineering-program) . Another 
example is Raytheon's MathMovesU program, which among others showcases math 
(and) in action as students design and experience their own thrill ride using math 
fundamentals ( http://mathalive.eom/raytheon-mathmovesu/#raytheon ). STEM Education 
attempts to transform the typical teacher-centered classroom by encouraging a curriculum 
that is driven by problem- solving, discovery, exploratory learning, and require students to 
actively engage a situation in order to find its solution (Fioriello, 2010). The above 
initiatives, though worthwhile, they are optional to schools and not available to schools 
nation-wide. The Next Generation Science Standards to be released in April 2013 sets the 
stage for the application of engineering design in K-12 science lesson nationwide. Draft 
II ( http://www.nextgenscience.org ) of the Next Generation Standards incorporates 
engineering design in science lessons in K-12 classrooms. Some of the well-known 
STEM approaches are P1BL, inquiry, problem-based learning and DBS. These 
approaches are however similar and intertwined. 
Relationship between P1BL, PjBL, DBS and Inquiry 

Problem-based learning, as it is generally known today, evolved from innovative 
health sciences curricula introduced in North America over 30 years ago at McMaster 
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University in Canada and became an accepted instructional approach in medical 
institutions across North America and in Europe in the 1990s (Boud and Feletti, 1997). 
Hmelo-Silver (2004) described Problem-based learning as an instructional method in 
which students learn through facilitated problem solving that centers on a complex 
problem, which does not have a single correct answer. Torp and Sage (2002) described 
Problem-based learning as focused, experiential learning organized around the 
investigation and resolution of messy, real-world problems. They described students in a 
Problem-based learning classroom as engaged problem solvers, seeking to identify the 
root problem and the conditions needed for a good solution and in the process becoming 
self-directed learners. Savery (2006) described problem-based learning as a learner- 
centered instructional approach that empowers learners to conduct research, integrate 
theory and practice, and apply knowledge and skills to develop a viable solution to a 
defined problem. The Problem-based Learning Institute has developed curricular 
materials and teacher-training programs for all core disciplines in high school (Barrows 
& Kelson, 1993). Problem-based learning is now used in multiple domains such as pre- 
service teacher education (Hmelo-Silver, 2004) and chemical engineering (Woods, 1994). 
Project-Based Learning (PjBL) has a long historical background (Grant, 2002). It 
was first discussed in W. Kilpatrick's article "The Project Method", published in 1918 
(Wrigley, 1998). Since, John Dewey's "problem solving" method began to resemble the 
traditional teaching method, W. Kilpatrick began to spread "The Project Method" 
(Oguzkan, 1989). P1BL can thus be said to have emerged as a synthesis of John Dewey's 
and Kilpatrick's views on learning. According to the definitions found in PjBL 
handbooks for teachers, PjBL involves complex tasks, based on challenging questions or 
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problems, that involve students in design, problem-solving, decision making, or 
investigative activities; give students the opportunity to work relatively autonomously 
over extended periods of time; and culminate in realistic products or presentations (Jones, 
Rasmus sen, & Moffitt, 1997; Thomas, Mergendoller, & Michaelson, 1999). 

Inquiry-based learning, just like PjBL is grounded in the philosophy of John 
Dewey, who believed that education begun with the curiosity of the learner. Inquiry- 
based learning is a student-centered, active learning approach focused on questioning, 
critical thinking, and problem solving (Savery, 2006). Inquiry-based learning activities 
begin with a question followed by investigating solutions, creating new knowledge as 
information is gathered and understood, discussing discoveries and experiences, and 
reflecting on new-found knowledge. Inquiry-based learning is frequently used in science 
education and encourages a hands-on approach where students practice the scientific 
method on authentic problems (or questions). 

DBS was designed around a stepwise description of the design process (Davis, 
Hawley, McMullan, & Spilka, 1997) and a social constructivist perspective of learning 
(Blumenfeld, Marx, Patrick, Krajcik, & Soloway, 1997). Silk, Schunn & Cary (2009) 
define design-based learning in general as a type of project based learning, which 
engages students in the process of developing, building, and evaluating a product they 
have designed. According to Krajcik, Blumenfeld, Marx, Bass, Fredricks, & Soloway 
(1998) DBS is an inquiry-based pedagogy that grew out of Project-Based Science, which 
is similar to Problem-based learning. Design-based science (DBS) is a science pedagogy 
that aligned with the goals of STEM education and inquiry-based science education. 
DBS, according to Fortus, Dershimer, Krajcik, Marx and Mamlok-Naaman (2005) is an 
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inquiry-based science pedagogy in which new scientific knowledge and problem-solving 
skills are constructed in the context of designing artifacts. In a PjBL environment learners 
are usually provided with specifications of a desired end product achieved by following 
correct procedure. However, learners are likely to encounter several problems that 
generate "teachable moments" (Lehman, George, Buchanan, & Rush, 2006). 

The relationship between PlbL, PjBL, DBS and IBL are presented in Figure 2. The 
primary difference between P1BL and inquiry-based learning relates to the role of the 
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tutor (Savery, 2006). In an inquiry-based approach the tutor is both a facilitator of 
learning (encouraging/expecting higher-order thinking) and a provider of information. In 
a PjBL approach the tutor supports the process and expects learners to make their 
thinking clear, but the tutor does not provide information related to the problem - that is 
the responsibility of the learners. DBS, however, incorporates inquiry and the design 
process. Common to these three approaches is certainly the presence a problem to be 
solved with the learner as an active player and the teacher as a facilitator, hence their 
inter-relatedness. It must be noted that the choice of DBS for the current study was made 
due to the promising results reported in previous research efforts in the areas of Learning 
by Design (Kolodner, Camp, Crismond, Fasse, Gray, Holbrool, Puntambekar, & Ryan, 
2003), project-based learning (Prince, 2004; Thomas, 2000) and problem-based learning 
(Akinoglu and Tandogan, 2007). The choice of DBS over other approaches was 
influenced by the fact that it combines inquiry and project-based approaches, both of 
which are consistent with problem solving as a goal of science education. 
Potential of DBS for Low-SES and Low-Performing Students 

The results of preliminary studies by Fortus, Dershimer, Krajcik, Marx and 
Mamlok-Naaman, (2005) and Puntambekar & Kolodner, 2005 imply that DBS and other 
inquiry-based pedagogies have the potential of helping students develop science 
knowledge. Engaging students in design-based learning or problem-based learning within 
a science classroom has the potential of helping students develop problem solving skills 
and scientific inquiry skills (Kolodner, Camp, Crismond, Fasse, Gray, Holbrool, 
Puntambekar, and Ryan, 2003; Silk, Schunn and Strand, 2007). In their study of the 
effects of DBS on science achievement among middle school students by gender, socio 
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economic status (SES) and race-ethnicity, Mehalik, Doppelt and Schuun (2008) reported 
that low-achieving African American students benefited the most from DBS. The above 
findings provide an excellent opportunity to study the effects of innovative pedagogies 
such as DBS on the improvement of science achievement gaps in secondary education. 
This is because they are primarily focused on middle schools. More research is thus 
needed in subjects such as Physics, Chemistry and Biology in high-school settings. 
Efforts to draw attention to the importance of problem solving in science are exemplified 
not only by the increasing number of project-/problem- solving programs but also by 
PISA. The potential of DBS in improving student achievement and problem solving 
competencies in science may emanate not only from its student-centered, hands-on 
approach but also from it contextualization of science. 

The effects of contextualization, such as provided by DBS were studied by 
Wisely (2009). He investigated the hypothesis that low- skilled students can learn more 
effectively and advance to college-level programs more readily through contextualization 
of basic skills instruction. The results of his study, which was however not in science, 
showed that participation in contextualization was associated with the completion of 
developmental education courses and the speed of entry into, and performance and 
completion of, college level courses. These positive effects were however, limited to non- 
white students: no effects for contextualization were found for white students. There is an 
interesting overlap between this and Bernstein's (1975) and Holland's (1981), finding 
that contextualizing science problems through real-world problem-based instruction, 
aligned with lower class students' preferred ways of thinking. This overlap is the 
relationship between race and socio-economic status: the idea that low skilled and lower- 
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class students benefited more from contextualization. Indeed, LaVeist (2005) describes 
the correlation between race and SES as substantial, with Whites and Pacific Asians 
having a high SES while African Americans and Hispanics tend to belong to the low SES 
group. According to Baker, Hope, and Karandjeff (2009), contextualization has been 
defined in numerous ways. For the purposes of this study I refer to the proposal by 
Mazzeo, Rab, and Alssid (2003) that, 

"Contextualization is a diverse family of instructional strategies designed to more 
seamlessly link the learning of foundational skills and academic or occupational 
content by focusing teaching and learning squarely on concrete applications in a 
specific context that is of interest to the student" (p. 3). 

The above findings therefore present another opportunity to investigate the effects 
of DBS on the development of problem-solving skills among various groups of students. 
If the relationship between DBS and science achievement among low-class African 
American students and the correlation between science literacy and problem solving 
skills are real then DBS may contribute to fighting the downward trend of minority 
students' involvement in STEM courses. 
PISA and Problem Solving 

In 2003, PISA measured students' problem solving capabilities. Problem solving 
will next be measured in 2012. The PISA 2003 problem- solving assessment measured the 
capability of fifteen year olds to apply knowledge to solving cross-disciplinary tasks, 
which approximate real-life situations. While U.S. students showed improved science 
results between 1995 and 2003 on their TIMSS average science score, their 2003 PISA 
average score in science and problem solving were below the international average 
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(National Science Board, 2006). This confirms the observation by OEDC (2004) that 
U.S. students solved problems at the basic level 

( http://www.oecd.org/dataoecd/25/12/34009000.pdf ). These students were consistently 
able to understand the nature of a problem and the relevant data associated with a 
problem's major features. However, most of these students were generally incapable of 
dealing with multi-faceted problems involving multiple data sources or requiring 
analytical reasoning with the information provided. Data from the PISA 2003 also reveals 
a high correlation of 0.8 between problem solving competency and science achievement. 

The PISA 2003 data also describe the relationship between international socio- 
economic index (largely determined by parental occupational status) and problem solving 
capabilities. Fifty percent of the variance in problem solving performance in the U.S. 
results was explained by international socio-economic index. The disparity in problem 
solving performance between the top and bottom socio-economic index for the U.S. is 
approximately 90 score points, close to one proficiency level in problem- solving 
performance (OECD, 2003). Improving this relationship between problem- solving 
performance and socio-economic status may therefore contribute significantly to closing 
the achievement gap between socio-economic classes. 

Earlier research studies by Clewell & Ginorio (1996); Creswell & Houston (1980) 
suggest that race-ethnicity and gender explain a significant portion of the variance in 
science achievement scores to varying degrees. If problem solving is significantly 
correlated to science achievement and SES, then any instructional approach that improves 
the problem solving competencies of students could provide leverage in closing 
achievement gaps between genders, SES and race. Also, on the premise that almost 
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everyone naturally engages in problem- solving (Nickerson, 1994) and design activities 
(Roberts, 1995; Baynes, 1994) it can be inferred that design-based lessons have the 
potential to address the basic capacity of all students. The current study therefore 
attempts to ascertain the impact of DBS on the development of problem-solving skills 
among students across gender, racial-ethnic and socio-economic status (SES). 

The integration of science, mathematics, and technology has been a common 
expectation in science education reform, to encourage problem solving, particularly in 
Project 2061. According to Project 2061, 

"Some important themes pervade science, mathematics, and technology . . . They 
are ideas that transcend disciplinary boundaries and prove fruitful in explanation, 
in theory, in observation, and in design" (AAAS, 1989, p. 155). 
It is therefore anticipated that science education will help students learn to integrate cross 
disciplinary principles and knowledge in solving problems. Numerous documents stress 
the importance of technology in science learning. Some of these efforts include: The 
National Science Education Standards [NSES] (National Research Council [NRC], 
1996), and Project 2061 (American Association for the Advancement of Science 
[AAAS], 1989, 1993). Technology is quite often used in a wide variety of meanings. 
However, among many definitions of technology, the above documents consistently use 
technology to refer to engineering, design, or engineering and design interchangeably 
(Raizen, Sellwood, Todd, & Vickers, 1995; Roth, 1998). The component of technology 
most closely allied to scientific inquiry and mathematical modeling is engineering. In its 
broadest sense, engineering consists of construing a problem and designing a solution for 
it. According to NSES, 
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"The central distinguishing characteristic between science and technology is a 
difference in goal: The goal of science is to understand the natural world, and the 
goal of technology is to make modifications in the world to meet human needs" 
(NRC, 1996, p. 24). 

The integration of science and technology thus allows students to use scientific 
knowledge to design and solve real- world problems. Despite its importance in making 
science relevant and practical in everyday life, technology as engineering and design has 
been largely ignored in school science (Raizen, Sellwood, Todd, & Vickers, 1995). 
However, the situation is changing. There have been increasing efforts by corporate 
bodies and the U.S. government to improve STEM programs in schools (Delaware Valley 
Industrial Resource Center and National Council for Advanced Manufacturing, 2006; 
PLTW, 2011; GAO, 2005). The result is an increasing popularity of pedagogies such as 
DBS, P1BL and PjBL, all of which have problem solving in common. These science 
instructional pedagogies (in science) enable the transfer of science knowledge to solving 
real- world problems. A number of empirical studies link real- world problem- solving 
associated pedagogies to science achievement. 
Statement of the Problem 

Student achievement in science (and math) in the United States has been on the 
decline over the past couple of decades, both nationally and internationally (Ornstein, 
2010; U. S. Department of Education 2004, 2006; NEAP, 2005), particularly in 
secondary education. The achievement gaps between males and females (NSF, 2008; 
Ornstein, 2010) and race (Ornstein, 2010; Clewell & Ginorio, 1996; Creswell & Houston, 
1980) have indeed not improved. An observable direct consequence of this decline is the 
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decreasing of competitive edge by the United States in the global market place and the 
leveling of college enrollment into STEM programs while the opposite is the case in 
other industrialized nations (Ornstein, 2010). The decline in student academic 
achievement in science (and math) has been attributed to a myriad of factors including 
pre-college educational preparation and high-school test scores in math and science 
(Griffith, 2010), family characteristics and educational support variables, attitudes toward 
math and science, and differences in aptitude (Benbow and Arjmand, 1990) and problem 
solving skills (Ornstein, 2010; OECD, 2003). 

Problem- solving associated science pedagogies such as DBS and P1BL have the 
potential of improving students' creativity/critical thinking, problem- solving ability, 
science-process skills and consequently science achievement (Mehalik, Doppelt and 
Schunn 2008; Fortus, Dershimer, Krajcik, Marx and Mamlok-Naaman, 2004, 2005; 
OECD, 2003; Chang, 2001a, 2001b). Other factors that have been verified to improve 
science achievement, particularly among African American students in particular, include 
their experience/conflict with science discourse (Brown, 2004; Fang, 2004; Bergin & 
Cooks, 2002). Systemic problems of teaching African- American students, including 
problems of teachers lacking science knowledge, poorly trained teachers, and poor 
expectations continue to exist and can be seen as a social inequity issue for African- 
American learners (Atwater, 2000) as well as other low-income urban students. 

The current study identifies real-world problem- solving ability for study for a 
number of reasons, namely, a) it has been a long-standing goal of science education, b) it 
has been identified as one of the 21 st century skills that will help students be successful in 
life and at their workplaces (Partnership for 21 st Century Skills, 2009), c) it correlates 
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highly with science achievement (OECD, 2003), d) a number of studies suggest that 
DBS, a project-based inquiry approach can improve real-world problem-solving skills as 
well as improve science achievement among one of the target groups of this study, 
namely, African Americans (Mehalik, Doppelt and Schuun, 2008; Kolodner, 2002; Rivet 
& Krajcik, 2004), and that e) boys and girls tend to favor hands-on activities (such as in 
DBS), which result in better science attitudes (Jovanic & Steinbach King, 1998). 
Rational/Purpose of the Study 

A number of factors have been identified to account for the decline in science 
achievement in the U.S. locally and on the international scene. Of these factors real-world 
problem- solving skills will be the focus of this study for reasons explained above. While 
there have been signs of improvement in science scores during the last few years, 
achievement gaps (between race/ethnicity, gender and SES) remain a concern. Some 
studies have identified the direct effect of DBS on science achievement among African 
American students, while observing a high correlation between real- world problem- 
solving skills and science achievement. The direct effect of real- world problem- solving 
skills and science achievement needs to be studied across race/ethnicity, gender and SES. 
Such a study will not only confirm the efficacy of DBS in helping close the achievement 
gap but also in preparing students both for academics and their future workplaces. 

Despite the findings stated earlier however, not enough studies have been done to 
demonstrate the effects of DBS on the development of real- world problem- solving skills 
across gender, racial-ethnic and SES and the consequent improvement in science 
achievement among high school students. Shepardson and Pizzini (1994), in one of a few 
such studies, reported that there is no significant difference in science achievement 
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between middle school boys and girls who learned science concepts through problem- 
solving approaches. There is the need to study this relationship among high school 
students where the gender gap widens even further. Similarly any differences in the effect 
of design-based learning across gender, racial-ethnic and socio-economic groups thus 
needs to be established and subsequently understood. 

The purpose of this study is thus to investigate the differences in the real- world 
problem- solving abilities (and science achievement) of high school students of different 
gender, race and SES after treatment with DBS in Traditional Chemistry Classes. The 
"effects" of DBS on these students' problem solving competency and science 
achievement will be described by group means, variances, correlation ratios, etc. Results 
from this study will contribute to responses to calls by Ault (1994) for research into the 
nature of learning and problem- solving in the area of science education, and by Thomas 
(2000) for more research on the effectiveness of P1BL. 
Research Questions 

In order to achieve the above objective the research questions this study seeks to 
address were: 

1. Does DBS have any effect on the problem solving competencies of students in a 

high school traditional chemistry class? 

2. Does the effect of DBS on problem solving competency depend on gender? 

3. Does the effect of DBS on problem solving competency depend on race? 

4. Does the effect of DBS on problem solving competency depend on SES? 
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5. Does DBS have any effect on the chemistry achievement of students in a high 

school traditional chemistry class? 

6. Does the effect of DBS on chemistry achievement vary depending on gender? 

7. Does the effect of DBS on chemistry achievement vary depending on race? 

8. Does the effect of DBS on chemistry achievement vary depending on SES? 

9. Is the problem solving competency of students in a traditional chemistry class 

predictive of their chemistry achievement? 

Definition of Terms 

Problem: A problem exists when there is an imbalance (referred to by Festinger, 1962 as 

"cognitive dissonance") between the concepts inherent in the problem situation and the 

conceptual schema of the individual, which motivates the individual to find a solution. 

Ill-structured problem: This is problem that addresses complex issues and thus cannot 

easily be described in a concise, complete manner. Furthermore, competing factors may 

suggest several approaches to the problem, requiring careful analysis to determine the 

best approach. 

Well-defined problem: A well-defined problem is identified by a testable goal state 

reachable from an initial state via one or more possible paths. 

Problem solving competency: This is an individual's capacity to use cognitive processes 

to confront and resolve real, cross-disciplinary situations where the solution path is not 

immediately obvious and where the literacy domains or curricular areas that might be 

applicable are not within a single domain of mathematics, science and other domains. 
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Design-Based Science (DBS): Design-Based Science is an inquiry-based project-based 
science pedagogy in which new scientific knowledge is constructed in the context of 
designing artifacts. 

Inquiry-based learning (IBL): A student-centered, active learning approach focused on 
questioning, critical thinking, and problem solving. It involves making observations, 
gathering, analyzing and interpreting data in an attempt to answer a question/problem 
through the scientific process 

Problem-Based Learning (P1BL): An instructional method in which students learn 
through facilitated problem solving that centers on a complex problem, which does not 
have a single correct answer. 

Project-based Learning (PjBL): In project-based learning learners are usually provided 
with specifications of a desired end product achieved by following correct problem- 
solving procedure. 

Contextualization: A diverse family of instructional strategies designed to utilize 
particular situations or events that occur outside of science class or are of particular 
interest to students to motivate and guide the presentation of science ideas and concepts. 
Contextualizing often takes the form of real- world examples or problems that are 
meaningful to students personally, to the local area, or to the scientific community. 
Technology: Technology as used in this work involves making modifications in the 
world to meet human needs. 

Real-world problem: an ill-defined problem that calls on individuals to merge 
knowledge and strategies to confront and resolve a problem readily identifiable as arising 
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from real-life. It is a situation where decisions are not clear-cut, requirements can 

conflict, and optimization rather than 'proof is needed. 

Traditional Chemistry: This is an introductory chemistry taught to students taking 

chemistry for the first time and usually as a science requirement for high school 

graduation. 
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CHAPTER II 
REVIEW OF THE LITERATURE 
Introduction 

In the following sections, a critical overview of the current state of research in the 
areas of design-based learning, and the development problem-solving skills are provided. 
The overview also includes trends in gender and race-ethnicity differences in science 
achievement. Another section is dedicated to an analysis of the research findings on the 
impact of project-based learning on students' science performance in terms of knowledge 
acquisition, attitudes and metacognitive development. A brief overview of the 
foundations and rationale for using DBS and the models of DBS are presented. Studies of 
the relationship between DBS and science achievement are elucidated. Theories of 
problem solving are reviewed as a lens through which the results of this study will be 
analyzed and interpreted. 
Foundations of Design-Based Learning: Constructivism 

DBS was developed over the course of the 1999-2000 school year, by the Center 
for Highly Interactive Computing in Education (hi-ce) at the University of Michigan. 
DBS (like other project-based pedagogies) is a detailed instructional model rooted in 
inquiry and which is consistent with the principles of instruction arising from 
constructivism (Savery & Duff, 1995; Krajcik, Czerniak, & Berger, 2002). A review of 
constructivism will therefore be helpful in order to appreciate the potential impact of 
DBS on knowledge acquisition and the development of problem- solving skills. 
Constructivism is a philosophical view about how people come to understand or know. 
Each of us builds our own key to knowing by making sense of the world. 
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Constructivist theory in education comes primarily from the work of John Dewey 
(1938) and Jean Piaget (1977). Working from the idea that learners construct their own 
knowledge, both Dewey and Piaget contended that the stimulus for learning is some 
experience of cognitive conflict, or "puzzlement" (Savery & Duffy, 1995). Dewey argued 
that learning should prepare a person for life, not simply for work. He proposed that 
learning should therefore be organized around the interests of the learner and that 
learning is an active effort by learners interested in resolving particular issues. Piaget, 
similarly, proposed that cognitive change and learning take place when a learner's way of 
thinking, or scheme, leads to puzzlement instead of producing what the learner expects. 
Such puzzlement then leads to accommodation (cognitive change) and a new sense of 
equilibrium. Learners bring their own suppositions to learning experiences based on what 
fits their experiences. 

Dewey's belief in competence in not only basic skills and personal qualities but 
also thinking skills — such as problem solving, reasoning, and knowing how to learn 
manifests in aspirations of recent educational policy directions (U.S. Department of 
Labor, 1991). During the Progressive Era, Dewey (1916) promoted the tackling of 
significant problems by students, as the ultimate way to engage learners in meaning- 
making and the development of problem solving ability. Dewey (1943) believed that 
learning should be situated within the context of the community. In this perception 
knowledge acquired is meaningful and relevant. Therefore, cognitive change often results 
from interactions with other learners who may hold different understandings (Volet, 
McGill & Pears, 1995). The associated social interactions may challenge learners' current 
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views as well as allow them to test their current understandings to see how well they help 
them make sense of and function in their world (Savery & Duffy, 1995). 

The idea that knowledge is constructed in the minds of learners has been 
extensively written about. For instance, Rorty (1991) described knowledge not as a 
representation of the real world or a "match" between perception and reality, but rather as 
a collection of conceptual structures that are adapted, or viable, within a person's range of 
experience. In other words, the person's knowledge "fits" with the world, much like how 
a key fits a lock (Bodner, 1986). 
Design-Based Science Framework 

The presumption associated with DBS is that students need opportunities to 
construct knowledge by solving problems through asking and refining questions; 
designing and conducting investigations; gathering, analyzing, and interpreting 
information and data; drawing conclusions; and reporting findings (Rivet & Krajcik, 
2004). In DBS, the design of the artifacts is not a culminating activity at the end of the 
curriculum, but rather it is the framework around which all the learning activities are 
organized. Any DBS instruction is characterized by five learning features (Singer, Marx, 
Krajcik, & Clay-Chambers, 2000). 

These five features are a) active construction, b) situated cognition, c) community, 
d) discourse, and e) cognitive tools. Students' active construction of knowledge refers to 
engaging students with the task in thought-demanding ways such as explaining, gathering 
evidence, generalizing, representing, and applying ideas (Perkins, 1993). Situated 
cognition refers to students making meaning through interactions between the world and 
others, and their interpretations of these interactions (Lave & Wenger, 1991) within the 
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contexts of the discipline. These interactions engage students with a community of 
practitioners in the discipline (Perkins, 1993). It enables students learn ways of knowing 
what counts as evidence, and how ideas are shared within the culture of the discipline. 
Participation also brings students into the language and discourse of the community of 
practice (Singer, Marx, Krajcik, & Clay-Chambers, 2000). Cognitive tools can extend 
what students can do and learn (Solomon & Perkins, 1989), in that they provide 
opportunities for students to visualize and explore phenomena that would not otherwise 
be possible in classrooms through manipulating multiple dynamic representations (Novak 
& Krajcik, 2005). 

The DBS learning cycle (Figure III) provides the framework for how classroom 
activities are structured. The cycle involves five stages. The first stage is 
contextualization. Context supplies significance for the tasks the students will be facing 
and provides trigger points for action - things the students can immediately begin to 
investigate (Kimbell, Stables & Green, 1996). The second stage involves background 
research, which can be in the form of searching and gathering relevant information, 
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benchmark lessons in which the teacher presents new scientific concepts, reading selected 
materials, sharing on a whiteboard of data collected in group experiments and then 
collectively analyzing the complete database, teacher-led demonstrations, computer- 
based simulations of relevant phenomena, and virtual expeditions to examine appropriate 
primary sources. 

During the third stage every student generates their solution to the design problem 
and presents it to their group members. The group decides which of the suggested 
solutions they prefer or they might combine the solutions. The group then writes a 
justification for their decision. In the fourth stage, each design team constructs a model or 
modifies an existing model based upon the design solution they decided upon in the 
former stage. For example, they might construct a three-dimensional model of a house or 
a cell phone antenna shield, or a cut-away drawing of an electrochemical cell. In the final 
stage, students' models are subjected to physical tests whenever possible, and they are 
presented to the entire class in a pin-up session (Kolodner, Stables, K., & Green, 1998; 
Schon, 1985). The models are laid out or hung up and the entire class moves from model 
to model, listening to the student-designers' descriptions and the teacher's comments, and 
offering their own critique. 
Review of Literature on Design-Based Science 

As the U.S. and other nations search for ways to close the achievement gap in 
STEM education in an effort to become more competitive in a global economy, the 
prospects of DBS should be seriously studied. Studies suggest that the adoption of DBS 
in science classrooms provide contextualized instruction, which benefits almost all 
students (Mehalik, Doppelt and Schuun, 2008; Rivet & Krajcik, 2004), particularly non- 
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white students as well as students from low SES families. The subsequent improvement 
in science achievement has been documented. The effect of DBS on the improvement of 
problem- solving skills, for the 21 st century, has also been suggested although more 
empirical studies need to be conducted. Some of these studies are presented in the 
following paragraphs. The incorporation of engineering design in the "Next Generation 
Science Standards" ( http://www.nextgenscience.org) , a national initiative for new 
science standards, is due to the potential of DBS improving science achievement and 
problem solving skills. 

One of the most recent studies on project-based inquiry instruction like DBS as a 
contextualizing instruction was conducted by Rivet & Krajcik (2004, 2008). Their study 
involved sixth through eighth-grade students within the Detroit Public School System, 
and investigated the effects of project-based curriculum materials on science achievement 
over a 10-week period. The curriculum materials used contextualized the learning of 
science in meaningful real-world problems and engage students in science inquiry. 
Students in these schools were representative of the district, which was over 91% African 
American, with over 70% of students receiving free or reduced-price lunches, and 85% of 
students were below grade level on standardized eighth-grade science assessment. The 
results show a strong and significant correlation between contextualizing score and all 
measures of learning. Similar suggestions have been made by other researchers, namely, 
calls for using "authentic tasks" (Lee & Songer, 2003), making science "relevant" 
(Fusco, 2001), and promoting community connections, and building from local contexts 
(Bouillion & Gomez, 2001). Also, Wisely (2009) investigated the hypothesis that low- 
skilled students can learn more effectively and advance to college-level programs more 
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readily through contextualization of basic skills instruction. The results of his study 
showed that participation in contextualization was associated with the completion of 
developmental education courses and the speed of entry into, and performance and 
completion of, college level courses. These positive effects were however, limited to non- 
white students; no effects for contextualization were found for white students. 

Studies that are specific to DBS, but which were not focused on contextualization 
are consistent with the above findings. For instance, Mehalik, Doppelt and Schuun (2008) 
contrasted overall performances and by gender, ethnicity, and socioeconomic status 
(SES) for middle school students learning science through traditional scripted inquiry 
versus a design-based inquiry (DBS). In their study the treatment group (DBS group) had 
a higher proportion of students from schools in the low SES range (53 percent of 587 
students versus 32 percent of 466 students in the control or scripted inquiry group). The 
four lowest SES schools in the district were in the DBS group. SES categories were based 
on the proportion of students considered by the district to be economically disadvantaged, 
with the low group having schools with more than 66 percent of their students 
economically disadvantaged. In terms of gender, the design group had a slightly higher 
proportion of female students (54 percent vs. 51 percent). As far as ethnicity, there was 
twice as much African American students (66 percent vs. 33 percent). The results 
suggested that the DBS approach for teaching science concepts had superior performance 
in terms of knowledge gain achievements in core science concepts, engagement, and 
retention when compared to a scripted inquiry approach. The DBS approach was most 
helpful to low-achieving African American students. 
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Since the 1970s, there has been an increasing emphasis on the use of the problem- 
solving approach in science teaching. Some research evidence has shown that explicit 
teaching of problem solving processes in science classes can improve students' problem- 
solving skills, students' cognitive development and science achievement (Huffman, 1997; 
Heyworth, 1998). Sternberg (1985) and Simon and Simon (1978) stressed that students 
meaningfully learn problem solving skills through concrete experiences. Visser (2002) 
compared the effects of problem based and lecture based teaching on student problem 
solving and attitudes in a high school genetics class. She found statistically significant 
differences (p<.05) in learning outcomes and motivation for students in the P1BL and 
Lecture/Discussion treatments. Problem- solving skills remain vital to the success of 
students even today (Partnership for 21 st Century Skills, 2009). 

Teachers can help students by providing explicit strategies that are procedurally 
structured to encourage students to become involved in their own learning and undertake 
the steps necessary to solve problems in science (Pizzini, Shepardson & Abell, 1989). It 
is suggested that DBS (and other project-based inquiry approaches) can improve problem 
solving skills (Forms, Dershimer, Krajcik, Marx and Mamlok-Naaman, 2005), however, 
the above studies do not show that the effects of DBS on science achievement is 
associated with a corresponding improvement in problem- solving skills, nor are there 
enough recent empirical studies to establish this relationship. In addition, majority of 
studies on DBS that compare student demographics have been limited to elementary and 
middle schools and not high schools where achievement gaps tend to be wider. The 
current study therefore seeks to investigate the link between DBS and real-world 
problem- solving skills of students in high school. 



40 



Theories of Problem Solving 

Any given problem has at least three components: the givens, goal(s) and 
operations. The Givens are the facts or pieces of information presented to describe the 
problem. The Goal(s) is the desired end state of the problem, while the Operations are the 
actions to be performed in order to reach the desired goal (Newell and Simon, 1972). To 
successfully solve a problem the problem solver's previous knowledge is important. 
Conceptually, there are two kinds of problem- solving knowledge (Gagne, Yekovich, and 
Yekovich, 1993): declarative knowledge, which is knowledge that something is the case, 
and procedural knowledge, which is knowledge of how to do something. While 
declarative knowledge is knowledge of facts, theories, events, and objects, procedural 
knowledge includes motor skills, cognitive skills, and cognitive strategies. Both 
declarative and procedural knowledge are activated in working memory as problem 
solving occurs. Declarative and procedural knowledge interact in a variety of ways 
during problem solving (Gagne, Yekovich, and Yekovich, 1993). 

There isn't only one way of solving problems. As a result there are a number of 
theories that may be used to explain how different problems may be solved. An overview 
of some of these theories and how they relate to the current study may provide some 
insight in understanding and perhaps explaining differences that may be observed 
between the different groups of students being investigated in this study. These theories 
include the a) constructivism, b) expert-novice theory, and c) cognitive theory. 
Constructivism 

Learners bring their own suppositions to learning experiences based on what fits 
their experiences. Thus, constructivism philosophy explains that knowledge is actively 
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constructed by an individual by comparing new ideas and concepts with their current 
knowledge (schema or mental models). Constructivist theory in education comes 
primarily from the work of John Dewey (1938) and Jean Piaget (1977). Dewey and 
Piaget contended that the stimulus for learning is some experience of cognitive conflict, 
or "puzzlement" (Savery & Duffy, 1995). Even the motivation to resolve the cognitive 
conflict may be spurred among others, by curiosity and personal or community needs. 
Hence, Dewey argued that learning should prepare a person for life, not simply for work. 
He proposed that learning should therefore be organized around the interests of the 
learner and that learning is an active effort by learners interested in resolving particular 
issues. 

During the Progressive Era, Dewey (1916) promoted the tackling of significant 
problems by students, as the ultimate way to engage learners in meaning-making and the 
development of problem solving ability. Dewey (1943) believed that learning should be 
situated within the context of the community. In this perception knowledge acquired is 
meaningful and relevant. Therefore, cognitive change often results from interactions with 
other learners who may hold different understandings (Volet, McGill & Pears, 1995) of a 
given situation. The associated social interactions may challenge learners' current views 
as well as allow them to test their current understandings to see how well they help them 
make sense of and function in their world (Savery & Duffy, 1995). Dewey's belief in 
competence in not only basic skills and personal qualities but also thinking skills — such 
as problem solving, reasoning, and knowing how to learn are encouraged by recent 
educational policy directions (U.S. Department of Labor, 1991). 
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In the current study students had to overcome specific constraints in order to meet 
needs that they had identified as relevant to their lives. In solving their problem, if the 
science concepts students were expected to learn were new to them, it was anticipated 
that students would tap into their prior knowledge (misconceptions or otherwise). If the 
problems were indeed relevant to their lives (as was the case), students would have the 
motivation to actively construct new knowledge as they solve the problem. 
Expert-Novice Theory 

A review of the differences between expert and novice problem solvers may help 
understand differences that may be observed during the current study. Three attributes are 
commonly used to differentiate expert from novice problem-solving characteristics 
(Muller, 1996). These attributes are a) conceptual understanding, b) basic, automated 
skills and c) domain-specific strategies. Conceptual understanding refers to both the 
actual information in memory and the organization of that information in memory. 
Conceptual understanding is closely related to schema theory, in which information is 
considered to be stored in memory as frameworks or structures that, once instantiated, 
provide a lens through which to view new information. Having a conceptual 
understanding of a domain means that an individual can make meaning of domain- 
specific situations or problems, based on prior knowledge of that domain. Students may 
approach a problem with this attribute already acquired or should develop it during the 
second or third stages of the DBS learning cycle: background research and the 
development of personal and group ideas. 

Basic, automated skills in any domain are those that allow an individual to 
perform necessary and routine operations without much thought. These skills are learned 
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to the extent that they become habitual and even unconscious, enabling individuals to 
operate quickly and accurately without over-burdening their short-term memories. This 
form of automaticity allows individuals to focus their attention on the more complex 
tasks associated with a specific domain and is a general attribute associated with experts 
in a domain. Automaticity supports the expert's speed and skill of execution. During a 
DBS unit students who do not already have such skills will need sufficient time to 
develop some automaticity, hence the importance of project duration. 

Unlike basic, automated skills, which occur unconsciously and thus do not tax 
short-term memory, domain- specific strategies remain under conscious control. They are 
the processes and procedures in a domain that an individual, even an expert, must 
consciously think about in order to solve a problem. They are, in other words, the 
procedural knowledge associated with a domain. Expert-novice differences have been 
studied and described within the context of these three attributes: Experts (a) exhibit 
better conceptual understanding of their domain; (b) use more automated skills and 
domain- specific strategies; and (c) have a conceptual understanding that is declarative, 
while basic skills and strategies are procedural (Miller, 1996). 

During the current study, in following the DBS framework, students are expected 
to develop conceptual understanding of the problem they intend to solve. It is also 
anticipated that students will develop an appreciation for the power of eliciting both 
declarative and automated skills in order to resolve a problem. In other words beyond the 
problem being resolved in the current study, students would recognize the need to look 
inward and elicit knowledge and skills required to solve any problem on hand. These 
students would move toward becoming expert problem solvers. 
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Cognitive Theory 

The cognitive theory is consistent with constructivism, in proposing that an 
inconsistency between behavior and beliefs motivates change. Cognitive psychologists, 
Wallas and Polya, separately developed four-stage models of problem solving. The four 
stages of problem solving identified by Wallas were: a) preparation - defining the 
problem and gathering information relevant to it, b) incubation - thinking about the 
problem at a subconscious level, c) inspiration - having a sudden insight into the solution 
of the problem, and d) verification - checking to be certain that the solution was correct 
(Ormrod, 1987). Polya's four steps in the problem-solving process included: a) 
understand the problem, b) devise a plan, c) carry out the plan, and (d) look backward 
(Ormrod, 1987). These two processes are very similar to each other and consistent with 
the five steps of the DBS learning cycle. Hence, assuming that the design challenge 
provides ample cognitive dissonance, the DBS learning cycle will provide the necessary 
paths needed by students in the current study to solve their problem of interest. It can 
therefore be anticipated that students who go through a unit by following the DBS 
learning cycle would ultimately become better problem solvers. 
Models of Problem Solving Instruction and Problem-Solving Skills 

Whether or not students' problem- solving skills will improve after a treatment 
such as the use of DBS units depends on students' basic knowledge of problem solving 
processes. In other words if a student does not have a clue how to approach the solution 
of real- world problems even a strategy that has a potential of improving problem-solving 
skills may not have a fair chance of successfully solving the problem. The selection of a 
problem solving model of instruction is however one of the critical choices a teacher 
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must make as s/he prepares students for real-world problem solving. Since the current 
study includes the provision of basic instruction in problem- solving strategies, a review 
of problem solving instruction models will provide a great reference for the model chosen 
and why. Three models commonly used are: a) Parnes' (1967) and Osborn's (1963) 
Creative Problem Solving (CPS) process, b) Bransford and Stein's (1984) Identify, 
Define, Explore, Act, and Look (IDEAL) model and, c) the Search, Solve, Create, and 
Share (SSCS) model created by Edward Pizzini (1987), and Pizzini, Abell, & Shepardson 
(1988). The CPS model is a hierarchical process, with each step depending on the 
preceding step. The five steps of the CPS model are: a) fact-finding, b) problem-finding, 
c) idea- finding, d) solution-finding, and e) acceptance-finding. 
The IDEAL and SSCS Models of Teaching Problem Solving 

The IDEAL model is also a five-step hierarchical model which involves: a) 
identifying the problem, b) defining and representing the problem, c) exploring 
alternative strategies, d) acting on the strategies, and e) looking back and evaluating the 
effects. The SSCS model on the other hand is a four step cyclical model allowing for re- 
entry into the various states of the model during the problem solving process (Figure IV). 
The SSCS model reduces the other problem 




Figure IV. The SSCS Problem Solving Cycle. 
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solving models into fewer steps; thereby, simplifying the process (Figure IV). 
Additionally, the SSCS model provides students with an opportunity to communicate 
their results (Figure V), something that is missing in other problem solving models of 
instruction. Although the SSCS model is not a pre-packaged curriculum, it can easily be 
incorporated into science instruction, providing a successful and creative way for students 
to learn science concepts and problem solving skills in science (Pizzini, Shepardson & 
Abell, 1989). The use of problem solving models in instruction calls for purposefulness 
on the part of the teacher and student. 

In any problem solving model of instruction the first level of learning includes 
problem recognition, the determination of information needed to solve the problem and 
where to obtain the information (Presseisen, 1985). Johnson, Ahlgren, Blout & Petit 
(1981) stressed the importance of how students search for an idea (concepts within the 
problem) that will assist them in understanding the problem. Glatthorn and Baron (1985) 
emphasized the importance of the search process, as well as setting goals, searching for 
possibilities, and evaluating evidence. Zoller (1987) suggested that the students' 
question-asking ability is an essential aspect of problem solving. Through the above 
processes, students derive meaning from the problem (Anderson & Smith, 1981; Winne 
& Mark, 1977) and can take ownership of the problem. Pizzini, Shepardson & Abel 
(1989) found that student ownership of the problem is one of the most essential variables 
resulting in successful problem solving. Providing students with the opportunity to select 
and pursue problems of concern and interest to them increases their motivation, 
persistence, and intensity to learn. 
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The SSCS model was developed on the premise that students meaningfully learn 
problem solving skills and science concepts through concrete experiences in solving 
problems in science, as evidenced by the literature. The SSCS model requires students to 
utilize various problem solving thinking skills identified by Sternberg (1985) and 
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Figure V. The SSCS Model as Related to the IDEAL and CPS Models (Pizzini, 
Shepardson & Abell, 1989) 

Presseisen (1985) (Figure VI). The four phases of the SSCS model according to Pizzini, 
Abell, & Shepardson (1988) are explained as follows. The Search phase of the SSCS 
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model involves brainstorming and other idea-generating techniques that facilitate the 
identification and development of researchable questions or problems in science. 
Demonstrations, magazines and newspaper articles, field trips, and science textbooks can 
lead students to the identification of researchable questions. In addition to identifying and 
developing questions and problems during the Search phase, students identify criteria for 
problem selection and state the question or problem in a researchable format. The Search 
phase assists students in relating the science concepts inherent in the problem to the 
relevant, existing science concepts embedded in their schema. This initiates the 
development of the problem space or mental representation of the problem. The problem 
then is identified and defined by the student, based on his/her existing conceptual 
schemata. 

The Solve phase requires students to generate and implement their plans for 
finding a solution to the problem they identified in the Search phase. During the Solve 
phase, the student reorganizes the concepts derived from the Search phase into a new 
"higher-order" that identifies the method for solving the problem and the desired solution, 
completing the development of the problem space. It is during the Solve phase that 
students apply operator(s) to solve the problem. If the operation is unable to solve the 
problem or creates an intermediate state, the student may re-enter the Search phase or 
continue to implement their plan (apply additional operators). The application of science 
concepts in the Solve phase provides meaning to the concepts as the student experiences 
the relationship between the concepts inherent in the problem, the concepts of the solved 
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Figure VI. Problem Solving/Thinking Skills within the SSCS Model (based on Sternberg, 
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problem, and the concepts applied to the problem, which are all linked to the students' 
conceptual schema. 

The Create phase requires students to create a product that relates to the 
problem/solution, compare the data to the problem, draw generalizations, and if necessary 
modify. Students employ skills such as reducing data to simpler levels of explanation or 
eliminating discrepancies. The Create phase enables students to evaluate their own 
thinking processes. The outcome of the Create phase is the development of an innovative 
product, which communicates the results of the Search and/or Solve phase to others. Self- 
evaluation (thinking about one's thinking) is the dominant activity throughout the Create 
phase. 

The basis of the Share phase is to involve students in communicating their 
problem solutions or question answers. The product created becomes the focus of the 
Share phase. The Share phase goes beyond simply communicating to students and others. 
Students articulate thinking through their communication and interaction, receive and 
process feedback, reflect on and evaluate solutions and answers, and generate potential 
Search questions. The generation of new potential Search questions occurs when an 
accepted solution creates a new problem, or when faulty reasoning or errors in the 
problem solving plan are discovered through external evaluation of the shared product. 
This enables the problem solver to identify problem solving skills which are in need of 
refinement, as well as initiate new Search questions. 
Framework for Assessing Problem Solving Skills 

The development of a problem- solving framework for assessing student 
performance is not easy. One of the reasons for this is that both individual and 
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collaborative problem solving are important for future learning, effective participation in 
society and for conducting personal activities. However, while measurement of individual 
problem solving competency may be achieved with greater ease, measurement of 
collaborative problem-solving competency is beset with numerous challenges (Reeff, 
Zabal & Blech, 2006). Foremost among these challenges are: how to a) assign credit to 
individual group members if this is required, b) account for differences across groups that 
may bias individual performance, and c) account for cultural differences in group 
dynamics. Most researchers who study problem solving in practice or research-based 
settings agree that in describing student problem solving, the major focus is on describing 
the cognitive acts students make in addressing, solving and reporting solutions (OECD 
2003). Cognitive acts therefore form the cornerstone of the PISA 2003 problem-solving 
assessment framework. 

Based on the definition of problem solving advanced earlier, the task to be 
performed by the student is shaped by its context(s), domain- specific knowledge and 
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strategies or skills required for solution. The PISA 2003 problem- solving assessment 
framework (Figure VII) adopted for the current study, include the following components: 
a) problem types, b) problem context, c) disciples involved, d) problem solving processes 
and e) reasoning skills. 

PISA 2003 chose to assess decision making, system analysis and design, and 
trouble- shooting as problem types because they are generic problem-solving structures 
that capture important aspects of everyday, real life analytical reasoning. They provide 
the structure within which problem solving is assessed without necessarily placing 
emphasis on domain knowledge but rather on the process and skills. Sample problems are 
provided in appendix B. Decision making problems enable one to determine whether a 
student understands different alternatives and constraints in a problem situation and if the 
student's decision satisfies the imposed constraints. A system analysis and design 
problem is different from a decision-making problem in the sense that the former, a) 
requires a student to analyze a system or design a solution to a problem instead of 
selecting from a set of alternatives, b) involves a complex system of interrelated variables 
and a solution is not clear-cut. Solution of system analysis and design problems requires 
the ability to identify the different variables and how they affect each other. In the case of 
designs such relationships must be considered in optimizing the desired goal. Trouble- 
shooting tasks on the other hand involve diagnosing, proposing a solution and sometimes 
executing this solution. Trouble-shooting requires a) an understanding of how a device or 
procedure works and b) the ability to identify the relevant features for the task and to 
create or apply a representation in order to successfully solve the problem at hand. In all 
the three types of problems a student's ability to evaluate, justify and communicate their 
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solution to another person(s), form integral aspects of the problem- solving competency. 
The PISA 2003 problem solving assessment involved problems embedded in real-life 
settings associated with personal life, work and leisure or community and society. 

While it may be necessary to identify the processes used by students as they solve 
problems the endeavor can be cumbersome if all problem types are considered. In the 
PISA 2003 framework problem solving processes considered are those based on 
cognitive analysis of the three types of problems assessed. The selection of these 
processes was informed by the work of cognitive psychologist such as Mayer & Wittrock 
(1996), Baxter & Glaser (1997) and Bransford, Brown & Cocking (1999) as well as by 
the seminal work of Polya (1945). These problem- solving processes include: a) 
understanding the problem, b) characterizing the problem, c) representing the problem, d) 
solving the problem, e) reflecting on the solution, and f) communicating the problem 
solution. The framework does not assume that these processes are hierarchical or even 
necessary for the solution of any problem. Indeed it recognizes that a student may solve a 
problem in a way that transcends the narrow linearity of the above model. In 
characterizing the problem students identify the variables in the problem and their 
interrelationships; decide relevant and irrelevant variables; construct hypotheses; and 
retrieve, organize, consider and critically evaluate contextual information. Representing 
the problem involves tabular, graphical, symbolic or verbal representation. Solving the 
problems, however, involves finding a solution that meets or exceeds the constraints and 
goals of the problem. Reflection involves examination and evaluation of the solution 
from different perspectives in an attempt to make it more acceptable as well as justifying 
these solutions. 
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The ability of a student to effectively use a given problem solving process does 
not only depend on his/her domain knowledge but also on the reasoning skills s/he 
possesses. In the PISA 2003 framework four reasoning skills are identified as being 
related to the problem types assessed. These skills include: a) analytical reasoning, which 
includes the application of principles from formal logic in determining cause-and-effect 
relations in order to select strategies, b) quantitative reasoning, which involve the ability 
to apply principles related to number sense and number operations, c) analogical 
reasoning, which involves the ability of the student to tap into his/her previous 
knowledge in order to venture into the unfamiliar, and d) combinatorial reasoning, which 
enables a problem solver to identify or rank all combinations of factors in order to 
achieve a set goal. The above mentioned skills are all expected to be demonstrated in 
SSCS, CPS and IDEAL framework (Figures IV and V). 

In summary, the PISA 2003 problem- solving assessment framework measures 
student problem- solving competency, using three types of problem: decision making, 
system analysis and design, and trouble shooting. The problems are cross -disciplinary 
and drawn from contexts that relate to student's personal life, work and leisure as well as 
to the community and society. Solution of a given problem may require the use of 
problem solving processes each of which depends of the reasoning skills of students. The 
problems are presented so that problem solving and not knowledge is being assessed and 
the problem solving processes don't necessarily have to be demonstrated at all and if 
demonstrated, not necessarily in any specific order. 
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CHAPTER III 
MATERIALS AND METHODS 
Introduction 

Chapter three presents the research methodology of the present study. It begins by 
summarizing the problem and restating the research questions. This is followed by a 
description of the subjects of the study and an overview or the research design. The 
chapter ends with data collection and data analysis procedures and statement of the 
hypotheses. 
Restatement of the Problem 

The purpose of this study is to investigate whether DBS affects student problem 
solving skills and science achievement across student demographics in a high school in 
Denver, Colorado. Effects of DBS will be studied across gender, race/ethnicity and SES 
among students in a traditional chemistry class. A strong correlation between problem- 
solving skills and science achievement has been reported (OECD, 2003). The effects of 
pedagogies such as project-based learning on science achievement and problem solving 
skills have not been extensively studied among different groups of students. For example, 
studies show that the use of pedagogies such as project-based learning, such as DBS, is 
associated with higher achievement in science in middle schools. However, more studies 
need to be conducted to investigate whether these pedagogies directly improve problem 
solving skills, with a consequent improvement in science achievement. With the 
achievement gaps in science not getting any better especially in high schools it will be 
helpful to establish the benefits of these pedagogies to high school students of different 
gender, race and SES. For the scope of this study DBS is chosen because of its potential 



56 



in improving science achievement among non- white students and its increasing presence 
in recent science education studies. 

The current study tries to achieve its purpose by attempting to answer the 
following research questions: 

1. Does DBS have any effect on the problem solving competencies of students in a 

high school traditional chemistry class? 

2. Does the effect of DBS on problem solving competency depend on gender? 

3. Does the effect of DBS on problem solving competency depend on race? 

4. Does the effect of DBS on problem solving competency depend on SES? 

5. Does DBS have any effect on the chemistry achievement of students in a high 

school traditional chemistry class? 

6. Does the effect of DBS on chemistry achievement vary depending on gender? 

7. Does the effect of DBS on chemistry achievement vary depending on race? 

8. Does the effect of DBS on chemistry achievement vary depending on SES? 

9. Is the problem solving competency of students in a traditional chemistry class 

predictive of their chemistry achievement? 

Hypothesis as Null Hypothesis 

The reformulation of the research questions as null hypotheses will facilitate the 
examination of the statistical analyses and are indicated as follows: 



57 



1. DBS has no effect on the problem solving competencies of students in a high 

school traditional chemistry class? 

2. The effect of DBS on problem solving competency does not depend on gender? 

3. The effect of DBS on problem solving competency does not depend on race? 

4. The effect of DBS on problem solving competency does not depend on SES? 

5. DBS has no effect on the chemistry achievement of students in a high school 

traditional chemistry class? 

6. The effect of DBS on chemistry achievement does not vary depending on gender? 

7. The effect of DBS on chemistry achievement does not vary depending on race? 

8. The effect of DBS on chemistry achievement does not vary depending on SES? 

9. Problem solving competency of students in a traditional chemistry class is not 

predictive of their chemistry achievement? 

Participants 

For the purposes of the current study, the treatment and control groups were 
derived through the non-probability means of purposive sampling. Krathwohl (2004) 
describes non-probable purposive sampling as a technique that is convenient to the 
researcher but which lessens questions about the representativeness of the sample. The 
subjects for this study included students in four traditional Chemistry classes in an urban 
high school in the State of Colorado. Majority of the students in these classes were low 
performing students, with a high proportion of African- American and Hispanic students. 
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Based on state tests (CSAP) these students were a fair representation of the African- 
American and Hispanic high-school student population in the state of Colorado. The 
Colorado Department of Education reports that in 201 1 60.4% of students from this high 
school in grades 9 and 10 were below proficient in science. Among the Black students 
while 89% were below proficient in math the number was 72% among Hispanics. Also, 
while 62% of Black students were below proficient in reading this number was 50% 
among Hispanic students. Only 23% of Black students were proficient in writing while 
30% of Hispanic students were proficient. With a student population of 1610 in 201 1, this 
school was 41.7% Black, 29.3% White, 23.3% Hispanic, 4.8% Asian and 0.9% American 
Indian. The percentage of students that was eligible for free and reduced lunch was 
51.8%. In contrast to the preceding statistics 67% of Asian students and 72% of White 
students in the school were at or above proficient in math. In reading 84% of White and 
Asian students were at or above proficient while in writing 81% Asian students and 78% 
of White students were at or above proficient. Four equivalent parallel traditional 
chemistry classes of ninety five (95) 10 l and ll l grade students were invited to 
participate in the study. Eighty two (82) students participated in this study. 

The treatment group comprised of 36 students (16 females and 20 males) while 
the control group was made up of 46 students (23 females and 23 males). The 
composition of the treatment group by race was as follows: Asians - 1; Black - 19; 
Hispanic - 14; Native American - 2; White - 0. The control group however consisted of: 
Asians - 1; Black - 21; Hispanic - 20; Native American - 1; White - 3. 
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Research Design 

A quasi-experimental pre/posttest research study with non-randomized sampling 
was conducted. Non-randomized assignment of participants to groups deals with intact 
groups and thus does not disrupt the existing research setting. This reduces the reactive 
effects of the experimental procedure and, therefore, improves the external validity of the 
design (Dimitrov & Rumrill, 2003). In order to ensure that any treatment and control 
group created were similar in their chemistry knowledge, district chemistry assessment 
data were reviewed before these groups were formed. The Denver Public Schools District 
chemistry assessment class average scores (ranging from 20% to 30%) showed that all 
four classes were similar. Subsequently two of the classes were randomly assigned to the 
treatment group while the other two classes are assigned to the control group. The 
treatment group learned anticipated chemistry concepts through DBS instruction 
(treatment) while the control group was expected to learn the chemistry concepts but 
through traditional methods of instruction. Pretests and posttests on problem solving 
competency and chemistry achievement were given to both groups to determine the effect 
of the treatment on problem solving competency and chemistry achievement. 

Problem solving competency was measured for both groups using the PISA 2003 
problem solving assessment protocol. After 2003 the next focus on PISA problem solving 
was 2012, after science in 2006 and reading in 2009 (OECD, 2003). At the time of this 
study the PISA 2012 problem solving items were not available, hence the use of the 2003 
problem solving items. Students' knowledge of chemistry concepts expected to be 
learned from the chemical energy/heating and cooling unit was also assessed before and 
after treatment. The unit included the three core chemical concepts: atomic interactions, 
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reactions and energy changes during reactions. In this design the method of instruction 
(DBS) was the independent variable. The dependent variables on the other hand were 
problem solving competency and knowledge of chemistry concepts (appendix F). Sample 
questions for the assessment of knowledge of chemistry concepts are shown in appendix 
E. Weekly assessment data were collected only as a way of monitoring student progress 
and to keep students focused. This is because DBS, like other forms of PjBL is self- 
directed and students are likely to be at different stages of learning during the project 
Procedures 
The Treatment 

Before the implementation of the unit in both groups the teacher elicited ideas and 
understanding currently held by students about the chemistry and design of heating and 
cooling systems. For meaningful learning to occur, instruction should begin with an 
exploration of learners' interpretations and understandings of the science concepts to be 
addressed (Taber 2003). The treatment provided opportunities for students to learn about 
the design process and presented the problem that needed to be resolved through 
scientific principles from the heating/cooling unit and the design process. The control 
group was neither engaged in the design process nor required to solve a design challenge 
problem. 

The treatment was the DBS-Heating/Cooling System (Chemical Energy) unit 
used by Apedoe, Reynolds, Ellefson and Schunn (2008). As a first step, the treatment 
group watched the video clips "Engineering Design Process" by NASA and "Bombing 
Hitler's Dams". As they watched the latter video, they were expected to document what 
they learned about the researcher's processes, personal qualities and why they thought he 
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was successful in establishing how Hitler's Dams were successfully destroyed during 
World War II by allied forces. They were expected to discuss if and how he used the 
engineering design process as described in the first video. This activity enabled students 
to become familiar with key design ideas such as needs, requirements and functional 
decomposition. It also provided a context for student work during the 12- week unit as 
well as helped students see how scientific principles and engineering design go hand-in- 
hand during the search for solution to a problem. Students in the control group did not 
watch the video on engineering design. However, they watched the video on bombing 
Hitler's dams and were required to write an essay on how science contributed to ending 
World War II. 

The second main step involved students in the treatment groups working in 
groups to brainstorm their needs for a heating and cooling system in their own lives. This 
opportunity was expected to create a personal motivation for the design work and made 
the topic relevant across ethnicity, gender, and other micro-cultures and helped students 
see the relevance of science and technology in their daily lives (Apedoe, Reynolds, 
Ellefson and Schunn, 2008). When groups had identified their needs they were then 
required to design a heating or cooling system that relies on chemical energy to meet the 
need(s), using concepts/knowledge from the unit. For example, they could create systems 
that would a) help keep them cool in the summer when they are playing sports, b) prevent 
them from having to sit on a cold toilet seat during cold weather and c) keep them cool 
when on a date and things start to "heat up". Students also had the option of designing 
and building a prototype of a toy. Students' work involved three aspects: a) planning the 
design, b) deciding and studying the chemical reactions that meet their specific needs as 
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well as how the reactions will be contained and c) making a three to five minute 
presentation to their classmates who represented the board of directors of a firm that was 
looking for ideas to invest in. During the next step students thought about other examples 
of heating and cooling systems from their everyday lives to consider the parts of these 
systems that make them work. The students did this to develop suggestions for solution to 
the problem on hand. Students in the control group were not assigned a design challenge. 
Instead, they learned chemistry concepts (appendix F) through traditional methods such 
as lecture, word problems and scripted inquiry. 

The identification of other heating and/or cooling systems was intended to help 
students understand that systems are made of subsystems, which in turn can be broken 
down in order to understand how they function. Subsystem decomposition is critical for 
engineering design (Bradshaw 1992; Ulrich and Eppinger 2004). While students 
suggested solutions to the problem on hand the teacher ensured that students stayed with 
the use of required chemistry concepts (appendix F). Although actual heating and cooling 
systems tend to have more than two subsystems, as a result of limited time for the unit 
and the emphasis on science concept learning students were encouraged to work towards 
a two-subsystem design: the reaction subsystem where energy is produced and a 
container subsystem, which manages the transfer of energy in the system. Students were 
to spend more time on the reaction and container subsystems stage than the other two 
(planning the design and presenting the design) since most of the chemistry concepts will 
be learned during this stage. 

Figure VIII summarizes the process of planning the design. To illustrate the above 
process one of the student projects is described as follows. After deciding to design a 
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toilet seat warmer the group planned their design by making drawings of their design. As 
part of their design they identified their reaction system, which would generate the heat 
for their warmer and their container system, which would comprise materials that will 
hold the chemical(s) and/or transfer the generated heat. During reaction I the group 
researched the source of chemical energy during chemical reactions. The goal was to 
understand the effect of molecular shape, size and bond type as well as other factors in 
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determining the amount of heat change in a chemical reaction. During reaction II the 
group applied knowledge gained in system I to generate chemical energy. For example, 
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the group investigated factors affecting the quantity of heat generated. The group then 
used properties of materials (metals, plastics, leather, etc.) to decide where chemicals 
would be stored as well as what materials to insulate and transmit heat. 

According to Apedoe, Reynolds, Ellefson and Schunn (2008), the reaction 
subsystem (Reactions I and II) and the container subsystem address different, chemistry 
concepts and as a result if students went through them more times they gained a deeper 
understanding of the relevant chemistry concepts. As shown in appendix F, each of the 
subsystems indeed addresses one big idea. During each lesson students engaged in 
activities that challenge them to work with one or two of these key concepts as they 
discuss them with their teacher. The lessons build upon each other and culminate in a 
lesson titled "Connecting to the Big Idea". 
The Design-Science Cycle 

Activities within each subsystem (Reactions I and II) are structured cyclically so 
that students move from design goals to science goals and back to design goals. This 
cycle, made room for whole class discussions, team activities, and individual activities 
intended to maximize the learning of science content as well as design and science 
processes. Thus, this cycle can be called the design-science cycle (or the learning cycle), 
similar to the legacy cycle (Brophy & Bransford 2001). Students started the design- 
science cycle (Figure IX) in the "Design" phase at the "Create Design" node by 
developing a design idea and trying it out in the "Evaluate Outcome" node. They 
discussed reasons for their outcomes as a class or in-group during "Generate Reasons". 
During this stage, students addressed questions such as: a) Was our design successful? b) 
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What factors were important for the success of the design? c) What factors may have 
influenced the failed 



..''' Design Create 

,-'"' Jlf Design ' 

/' ^>^~^"~jr' " — "--- ^Evaluate 
,y Connect to **"~^,Outcome 

Big Ideas "'V 



Big Ideas "^A 

/ t \ 

f Public Dialogue ▼ 

K 1 Gen 

\ \ Generalize Ftei 

\ \, Results / 

vv — J 

*S Ani 



J 



Analyze "' : ' 

Results <^ 

Science 



Figure IX. The Design-Science Cycle (Apedoe, Reynolds, Ellefson and Schunn, 2008) 

performance of the design? Students proposed ways to systematically test some of their 
generated reasons and conduct these tests during "Test Ideas". Students then analyzed the 
results from their experiments and discussed their findings as a class/group during 
"Generalize Results" to uncover a pattern, theory, or trend. Finally, students arrived at 
"Connect to Big Idea", where they linked their design to the key science concept(s) that 
can be used to improve its performance. 

Overall, the Design-Science Cycle is structured to maintain a motivating design 
storyline while preventing students from wasting time floundering and encouraging them 
to focus attention on the selected core concepts. At the end of the reaction subsystem 
students built on their ideas developed by considering the material(s) for the container, 
and the properties such as density, thermal conductivity, melting point, specific heat 
capacity, etc. of the material. Student-groups presented their design to the class in ways 
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they found convenient and effective. Appendix G summarizes the activities for the 
treatment and control groups. While students in the treatment group learned about heating 
and cooling through a DBS unit over a 12-week period (Apedoe, Reynolds, Ellefson and 
Schunn 2008) the control group learned about heating and cooling through lecture, word 
problems and scripted inquiry, where students were given steps and guidelines to follow 
to complete the experiments. 
Data Sources 

Student biographical information (gender, race/ethnicity, SES) was the first set of 
information to be collected, before the start of the unit, by means of a survey from both 
groups. Students' gender and ethnicity was obtained from the school while SES was 
determined using the BSMSS (Appendix A). The BSMSS incorporates a student's 
parent's educational attainment and occupational prestige as well as his/her own 
educational attainment and occupational prestige (Barratt, 2006). Pretests of problem 
solving ability and chemistry concept knowledge in heating and cooling were 
administered to both the control and treatment groups. A sample of real- world PISA 2003 
problem- solving items, that were used for the pre-test and post-test can be found in 
Appendix B. The problem solving items assess problem solving competency in the 
following areas: decision making, system design and analysis, and troubleshooting. The 
chemistry concepts inventory is a twenty two item test that assesses understanding in 
chemical and physical changes. 
Teacher as Researcher 

Whenever the teacher of a given class is also a researcher in the same class there 
are some advantages and disadvantages. An advantage of a teacher-as-researcher 
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arrangement is the fact that the research environment is kept intact. The teacher-student 
and student-students relationships in the room are not interfered with. Students may more 
likely be open to sharing their challenges and inclinations with the teacher than an 
outsider. Another advantage is that the teacher-researcher usually has a long-term 
experience of the setting being studied and therefore know the history and information 
needed to understand what is going on in the setting (Hammersley, 1993). On the other 
hand, there is a risk of the teacher-researcher being biased in his/her judgments towards 
students and different situations. Therefore, in order to ensure inter-rater reliability and 
minimize any biases that may result from the teacher as researcher two peer observers 
were requested to visit two class periods each. They also reviewed sample student 
responses (problem solving assessment and Chemistry Concepts test) graded by the class 
teacher (researcher). 
Instrumentation 

The independent variable to be studied is participation in DBS unit (across 
gender, race and SES). The instruments that were used for data collection from the 
treatment and control groups are described below. The socioeconomic status of students 
was measured using the Barratt Simplified Measure of Social Status (BSMSS) shown in 
Appendix A. The BSMSS is a version of the A. B. Hollingshead (1975) four-factor index 
of social status, with a reliability of 0.85 and demonstrated to be a valid measure of SES 
(Cirino, Chin, Sevcik, Wolf, Lovett & Morris, 2002). The BSMSS accounts for an 
individual's parent's educational attainment and occupational prestige and combines them 
with the individual's own educational attainment and occupational prestige (Barratt, 
2006). BSMSS scores range from 8 (lowest SES) to 66 (highest SES). This measure of 
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SES is more likely to be accurate than student eligibility for free or reduced lunch since 
students living in single-parent families may have an unfair advantage of being classified 
as belonging to free or reduced lunch program while support from the other parent may 
not be considered in determining eligibility. The score that results from the BSMSS is 
ordinal only and is sufficient for regression analysis or for creating SES groups based on 
the data collected. 

In order to answer the first research question, the effect of DBS on the problem 
solving competency of a student was measured by administering a pretest and posttest 
using the PISA 2003 problem solving sample problems (Appendix B). The problems that 
were used in assessing problem solving competency included 19 items as in PISA 2003: 
7 decision making items, 7 system analysis and design items and 5 trouble shooting 
items. In this assessment, more emphasis is placed on decision-making followed by 
system design, with troubleshooting being allocated the least scores. Across the three 
problem types more difficult problems are scored at the middle of the scale (Appendix 
D). Some questions require students to construct their own responses, either by providing 
a brief answer from a wide range of possible answers (short-response items) or by 
constructing a longer response (open-constructed response items), allowing for the 
possibility of divergent, individual responses and opposing viewpoints. Other parts of the 
test are based on students constructing their own responses, but based on a very limited 
range of possible responses (closed-constructed response items), which are scored as 
either correct or incorrect. The remaining items are asked in multiple-choice format, in 
which students either make one choice from among four or five given alternatives 
(multiple-choice items) or a series of choices by circling a word or short phrase (for 
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example "yes" or "no") for each point of credit (complex multiple-choice items). The 
reliability of these assessments is 0.87 (OECD, 2005). 

The features of each problem type (goals, processes and sources of complexity) 
compared in Appendix C, serve as the basis for establishing a scale to describe increasing 
student proficiency in problem solving. The problem solving items would be scored using 
the rubric associated with each item (Appendix B), while the skill level of students would 
be determined using the PISA 2003 problem solving scale (Appendix D). The PISA 
problem-solving scale provides a representation of students' capacity to understand, 
characterize, represent, solve, reflect on and communicate their solutions to a problem. 
The total student problem solving score is use to determine the student's level (or 
competency) of problem solving. The three levels of proficiency in problem solving are: 
a) level 1 - basic problem solvers, b) level 2 - reasoning, decision-making problem 
solvers, and c) level 3 - reflective, communicative problem solvers. 

Level 3 students score above 592 points on the PISA problem solving Scale. 
Typically, these students are able to analyze a situation and make decisions, as well as 
think about the underlying relationships in a problem and relate them to the solution. 
These students are systematic problem solvers and construct their own representations to 
help them solve problems. They verify that their solution satisfies all requirements of the 
problem. These students communicate their solutions using accurate written statements 
and other representations. Students at the top of Level 3 can cope with multiple 
interrelated conditions that require them to work back and forth between their solution 
and the conditions laid out in the problem. 
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Students at Level 3 are also expected to be able to successfully complete tasks located at 
lower levels of the PISA problem- solving scale. The Students at Level 3 therefore 
possess the following skills: monitoring variables, accounting for temporal restrictions, 
and other constraints; troubleshooting, analytical, decision making, visualization, 
evaluation of their solution, effective handing of the complexity of multiple interrelated 
conditions and effective communication. These are problem solving skills that are 
associated with all three levels (metacognitive, performance and knowledge acquisition) 
of the SSCS Model (Figure VI). 

Students proficient at Level 2 score between 499 to 592 points on the problem- 
solving Scale. These students use reasoning and analytic processes and solve problems 
requiring decision making skills. They can apply various types of reasoning (inductive 
and deductive reasoning, reasoning about causes and effects, or reasoning with many 
combinations, which involves systematically comparing all possible variations in well- 
described situations) to analyze situations and to solve problems that require them to 
make a decision among well-defined alternatives. To analyze a system or make decisions, 
students at Level 2 combine and synthesize information from a variety of sources. They 
are able to combine various forms of representations 

(e.g. a formalized language, numerical information, and graphical information), handle 
unfamiliar representations (e.g. statements in a programming language or flow diagrams 
related to a mechanical or structural arrangement of components) and draw inferences 
based on two or more sources of information. Students at Level 2 are also expected to be 
able to successfully complete tasks located at Level 1 of the PISA problem-solving scale. 
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Students proficient at Level 1 score between 405 to 499 points on the problem- 
solving scale. They typically solve problems where they have to deal with only a single 
data source containing discrete, well-defined information. They understand the nature of 
a problem and consistently locate and retrieve information related to the major features of 
the problem. Students at Level 1 are able to transform the information in the problem to 
present the problem differently, e.g. take information from a table to create a drawing or 
graph. Also, students can apply information to check a limited number of well-defined 
conditions within the problem. However, students at Level 1 do not typically deal 
successfully with multi-faceted problems involving more than one data source or 
requiring them to reason with the information provided. Students below level 1, with 
scores of less than 405 points, are weak or emergent problem solvers. They consistently 
fail to understand even the easiest items in the assessment or fail to apply the necessary 
processes to characterize important features or represent the problems. At most, they can 
deal with straightforward problems with carefully structured tasks that require the 
students to give responses based on facts or to make observations with few or no 
inferences. They have significant difficulties in making decisions, analyzing or evaluating 
systems, and trouble- shooting situations. 

The effects of DBS on student proficiency in the science concepts and knowledge 
gain intended by the Heating and Cooling System unit would be assessed by 24 questions 
taken from the Chemical Concept Inventory (CCI) (American Chemical Society, 2001) 
and the American Chemical Society's (ACS) Test Item Bank for high school chemistry 
(Eubanks an Eubanks, 1993). Samples of these questions are shown in appendix E. The 
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reliability of CCI is 0.71 with proven validity (Krause, Birk, Bauer, Jenkins, & Pavelich, 

2004). 

Data Analysis 

In order to answer research questions one to eight, data collected were analyzed 
by ANCOVA, controlling for pretest scores. ANCOVA was used because it yields more 
powerful results, meaning there was a higher probability of finding group differences if 
indeed any difference existed and is not associated with an inflated a-level of significance 
(Dimitrov & Rumrill, 2003). Other analyses that could have been performed, namely: a) 
Analysis of variance (ANOVA) on gain scores, b) ANOVA on residual scores, or (d) 
Repeated measures ANOVA. The use of pretest scores in these methods helps to reduce 
error variance, thus producing more powerful tests than designs with no pretest data 
(Stevens, 1996). ANOVA on residual scores was not used because: a) when the residuals 
are obtained from the pooled within-group regression coefficients, ANOVA on residual 
scores results in an inflated a-level of significance and b) when the regression coefficient 
for the total sample of all groups combined is used, ANOVA on residual scores yields an 
inappropriately conservative test (Maxwell, Delaney, & Manheimer, 1985). Also, 
repeated measures ANOVA was not used because according to Huck & McLean (1975) 
and Jennings (1998) the results provided by repeated measures ANOVA for pretest- 
posttest data can be misleading. Specifically, the F test for the treatment main effect 
(which is of primary interest) is very conservative because the pretest scores are not 
affected by the treatment. Hence for the analysis of pretest-posttest designs Dimitrov & 
Rumrill (2003) recommend one-way ANOVA on gain scores or, even better, ANCOVA 
with the pretest scores as a covariate. 
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Analysis of Covariance (ANCOVA), was therefore performed (controlling for 
pretest scores) to determine if groups (including gender, race-ethnicity and SES) differed 
significantly in gains in problem solving competency and chemistry achievement in 
Chemical heating/Cooling, pursuant to the administration of the DBS instruction. For the 
ANCOVA procedure, the effect sizes are noted and discussed using the Eta value. The 
Eta value is the proportion of variation in the dependent variable (problem solving 
competency or Chemistry concept) that is attributable to the DBS instruction and other 
factors such as gender, race and SES. The dependent variables (within-group) in the study 
were: a) Problem solving competency, and b) Chemistry achievement, considered one at 
a time. Both of these are scale quantities. The main independent variable (between-group) 
is the treatment (with 2 levels - treatment and control). The other predictors are: gender, 
race and SES. All predictors were nominal variables. 

The third research question sought to investigate whether the problem solving 
competency of a student was predictive of his/her achievement in chemistry. This 
question was answered by performing a correlation for all participants to determine if 
students who did well on the problem solving assessment also performed well on the 
chemistry concepts inventory and vice versa. 
Summary 

The research methodology of the study involved a quasi-experimental 
pretest/posttest design with a non-randomized sample, comprising 10th and ll l grade 
students in four traditional chemistry classes. Two main groups were compared, namely 
treatment and control groups. The treatment was DBS instruction on chemical 
heating/cooling unit. The study sought to investigate: a) the effects of DBS on problem 
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solving competency across gender, race and SES, b) the effects of DBS on chemistry 
achievement across gender, race and SES, and c) whether problem solving competency is 
predictive of chemistry achievement. Analysis of Covariance (ANCOVA) was the 
preferred analysis due to its reduction of systemic bias caused by group differences in 
pretest scores and the higher power associated with its results. 

The generalizability of the findings of this study is restricted to: a) students of 
traditional chemistry who are taking chemistry for the first time, b) students who are 
generally not highly motivated to study chemistry and c) schools in an urban school 
district with similar student characteristics as the one used in this study. This is because 
testing one school makes generalization difficult since the individual school tested may 
generate better or worse results for students using that particular educational instruction. 
The students in one school may be from a completely different socioeconomic 
background or culture and therefore cannot be a representative sample of the population. 
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CHAPTER IV 
DATA ANALYSIS AND RESULTS 
Introduction 

The purpose of this study was to investigate the effects of DBS instruction on the 
problem solving competency and the concepts of chemical energy among different 
groups of students. Student groups used in the analysis included gender and race. The 
relationship between SES and problem solving competency was also considered. A 
treatment group of 33 students were taught a twelve- week chemical energy unit through 
lessons in which students designed solutions to real-world problems as well as design 
their own investigations. The study sought to answer the following questions: "Do the 
effects of DBS on problem solving competencies of students in a high school Traditional 
Chemistry class vary depending on gender, race and/or SES?"; "Do the effects of DBS on 
science achievement of students in a high school Traditional Chemistry class vary 
depending on gender, race and/or SES?" and "Do students who show improved real- 
world problem solving skills also perform well on science achievement?". A control 
group of 41 students studied the same unit in traditional classroom settings. This chapter 
presents results from the quantitative methods used in the study. 

To investigate research questions one to eight an Analysis of Covariance 
(ANCOVA) procedure was performed, using pretest scores as covariate, to compare the 
means between the treatment and control groups. The analysis was conducted to ascertain 
group differences attributable to DBS. For the ANCOVA procedure, the effect sizes are 

9 9 

noted and discussed using the Eta value. The Eta value is the proportion of variation in 
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the dependent variable (problem solving competency or Chemistry concept) that is 
attributable to the DBS instruction and other factors such as gender, race and SES. 

Data Analysis and Results 

Effects of DBS on Problem Solving Competency across Gender, Race and SES 

Research question one: Comparison of treatment and control groups. 

The first research question is as follow: Does DBS have any effect on problem 
solving competencies of students in a high school traditional chemistry class? In order to 
answer the first part of this question, an ANCOVA was conducted to determine if there 
were significant differences between the means problem solving competency scores the 
treatment and control groups. The results show that there was a statistically significant 
difference between the treatment and control groups in their problem solving competency 
after the treatment (DBS instruction). This significant difference was observed across all 
three aspects of problem solving measured in the study. 

The ANCOVA results comparing the mean total problem solving scores of the 
treatment and control groups are presented as follow. The following assumptions were 
tested, a) independence of observations, b) normal distribution of the dependent variable, 
c) homogeneity of variance, d) linear relationships between the covariate and dependent 
variable, and e) homogeneity of regression slopes. All assumptions were met. The results 
indicate that after controlling for the pretest scores on problem solving competency, there 
was a significant difference between the treatment and control groups in problem solving 
competency. Table II shows that the problem solving competency of the treatment and 
control groups were significantly different, F (1, 71) = 32.90, p < .001, eta = .32. Thus 
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32 percent of the variance in problem solving scores is explained by the treatment. Table 
I presents the means and standard deviations for the two groups on problem solving 
competency, before and after controlling for problem solving pretest scores. 

Table I 

Adjusted and Unadjusted Group Means and Variability for Problem Solving Competency 
Using Problem Solving Pretest Scores as Covariate 

Unadjusted Adjusted 



N M SD M SE 

Control group 41 3214.07 2056.72 2831.31 285.93 

Treatment group 33 5139.88 1934.76 5305.72 313.18 

Table II 

Analysis of Covariance for Problem Solving Competency as a Function of Group, Using 
Problem Solving Pretest Scores as Covariate 



Source df MS F p eta 2 



Problem solving pretest 1 47604131.04 21.90 < .001 .30 

Group 1 52939811.61 24.35 < .001 .32 

Error 71 2174182.62 
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Estimated Marginal Means of Total Prob. Solving Score (Posttest) 




Group - Control group 1 - Treatment group 

Covariates appearing in the model are evaluated at the fallowing values: Total Prob. Solving score (Pretest) 

= 2329.23 

Figure X. Mean Problem Solving Scores for Treatment and Control Groups 



Comparison of Mean Problem Solving Competency Scores 
Pretest and Posttest 

Total Prob. Solving score 
(Pretest) 
_ Total Prob. Solving Score 
(Posttest) 




Group - Control group 1 - Treatment group 
Figure XL Mean Problem Solving Scores (Pretest and Posttest) for Treatment and Control Groups 

Figures X and XI depict the effect of DBS in improving overall problem solving 
competency. Figure X is compares the estimated mean problem solving competency 
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scores of the control and treatment groups with pretest scores adjusted (controlled for). 
Figure XI on the other hand compares the pre/posttest mean problem solving competency 
scores of the control and treatment group. The latter graph shows that while the pretest 
scores were similar for the two groups, the posttest scores for the treatment group are 
much higher than those of the control group. Thus the DBS instruction appeared to have 
affected the problem solving competency of students. The problem solving items 
comprised of three types of problems, namely, decision-making (DM), system analysis 
and design (SAD), and troubleshooting (TS). 

It may be necessary to dig deeper to determine whether the above effect of DBS 
on problem solving competency was evident in all three aspects of problem solving. The 
ANCOVA results for the subtotals for each of the three areas of problem solving 
competency measured, controlling for pretest scores for each aspect of problem solving, 
indicated statistically significant differences between the treatment and control groups. 
For decision making competency, F (1, 71) = 18.47, p < .001, eta = .21. Table III 
presents the mean and standard deviations for the control and treatment groups on 
decision making competency before and after controlling for decision making pretest 
scores. 
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Table III 

Adjusted and Unadjusted Group Means and Variability for Decision Making Competency 
Using Decision Making Pretest Scores as Covariate 

Unadjusted Adjusted 



N M SD M SE 

Control group 41 1426.37 988.90 1415.57 108.62 

Treatment group 33 2101.36 766.98 2114.78 121.06 

Table IV 

Analysis of Covariance for Problem Solving Competency as a Function of Group, Using 
Decision Making Pretest Scores as Covariate 



Source df MS F p eta 2 



Decision making pretest 1 23603294.30 48.81 < .001 .41 

Group 1 8934757.71 18.47 < .001 .21 

Error 71 483627.45 

The ANCOVA results for system analysis competency, show that F (1, 71) = 
1 1.52, p < .05, eta 2 = .14. Table V presents the mean and standard deviations for the 



control and treatment groups on system analysis and design competency before and after 
controlling for system analysis and design pretest scores. 

Table V 

Adjusted and Unadjusted Group Means and Variability for System Analysis Competency 
Using System Analysis and Design Pretest Scores as Covariate 



N 



Unadjusted 



M 



SD 



Adjusted 



M 



SE 



Control group 41 



Treatment group 33 



1326.68 



1937.73 



1020.32 



986.69 



1310.07 127.55 



1953.37 142.18 



Table VI 

Analysis of Covariance for System Analysis Competency as a Function of Group, Using 
System Analysis and Analysis Pretest Scores as Covariate 



Source 



df 



MS 



eta 



System Analysis pretest 1 25458677.26 38.19 < .001 .35 



Group 
Error 



71 



7677008.59 11.52 < .05 .14 



666715.50 



82 



The ANCOVA results for troubleshooting also show a significant difference 
between the treatment and control groups for troubleshooting competency, F (1, 71) = 
15.74, p < .001, eta = .18. The assumption of homogeneity of variances was violated. 
However, because the cell sizes (41 and 33) were similar this violation did not present an 
issue (Leech, Barrett and Morgan, 2008). Table VII presents the mean and standard 
deviations for the control and treatment groups on troubleshooting competency before 
and after controlling for troubleshooting pretest scores. 



Table VII 

Adjusted and Unadjusted Group Means and Variability for System Analysis Competency 
Using Troubleshooting Pretest Scores as Covariate 

Unadjusted Adjusted 



N M SD M SE 

Control group 41 497.10 549.66 478.80 102.50 

Treatment group 33 1076.42 749.82 1099.16 114.67 
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Table VIII 

Analysis of Covariance for System Analysis Competency as a Function of Group, Using 
Troubleshooting Pretest Scores as Covariate 



Source of MS F p eta 2 



Troubleshooting pretest 1 433749.75 1.04 .30 .014 

Group 1 6570140.63 15.74 < .001 .18 

Error 71 417497.77 

The above analysis suggests that DBS improves all three aspects of problem 
solving competency measured in the study. This, thus provides an answer to research 
question one, thereby disproving the null hypothesis that the DBS has no effect on 
problem solving competency. 

Research question two: problem solving competency across gender. 

The preceding section suggests that there is a significant effect of DBS on 
problem solving competency. This section is dedicated to analyzing the data for gender 
differences. To search for any significant gender differences an ANCOVA was conducted 
to determine whether there were differences between the average total problem solving 
scores of male and female students in the treatment and control groups, after controlling 
for problem solving pretest scores. Group and gender were used as fixed factors. The 
following assumptions were once again tested, a) independence of observations, b) 
normal distribution of the dependent variable, c) homogeneity of variance, d) linear 



84 



relationships between the covariate and dependent variable, and e) homogeneity of 
regression slopes. All the assumptions were met. The results indicate that the problem 
solving competencies of the male and female students in the treatment group on the one 
hand and the control group on the other were significantly different, F (1, 51) = 5.58, p < 
.05, eta = .099. The means and standard deviations for the two groups are presented in 
Table IX. 

Table IX 

Adjusted and Unadjusted Gender Means and Variability for Problem Solving 
Competency Using Problem Solving Pretest Scores as Covariate 

Unadjusted Adjusted 



N M SD M SE 

Control group 41 3214.07 2056.72 2831.31 285.93 

Female 22 2882.09 2066.21 2510.45 418.31 

Male 19 3598.47 2031.94 3216.34 377.05 

Treatment group 33 5139.88 1934.70 5305.72 313.18 

Female 15 5731.07 2069.17 5802.91 446.13 

Male 18 4647.22 1718.98 4709.09 443.54 
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Table X 

Analysis of Covariance for Problem Solving Competency as a Function of Gender, Using 
Problem Solving Pretest Scores as Covariate 



Source 



df 



MS 



eta 



Problem solving pretest 1 



Gender 



47604131.04 21.90 < .001 .30 



167923.86 .08 



.782 .00 



Group*Gender 
Error 



51 



12121040.13 5.58 



2174182.62 



<.05 .10 



The treatment (DBS) therefore explained about ten percent of the variance in female and 
male problem solving scores between the treatment and control groups. Figure XI shows 
that both males and females benefited from the treatment, with females in the treatment 
group a slight edge over males. Males in the control group performed better than females 
on the problem solving competency protocol. The interaction therefore indicates a small 
benefit of DBS for female students over male students on problem solving competency 
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Estimated Marginal Means of Total Prob. Solving Score (Posttest) 



Gender 

- Female 

1 - Male 




Group (0 - Control group 1 - Treatment group) 

Covariates appearing in the model are evaluated at the following values: Total Prob. Solving score ("Pretest) 

= 2329.23 



Figure XII. Effects of DBS on Problem Solving Competency across Gender 

The ANCOVA results for the previous section indicate a significant 
difference between the treatment and control groups in problem solving competency. 
Thus, DBS appeared to have improved the problem solving competency of students. In 
an attempt to answer research question one, this section sought to investigate if this effect 
of DBS was significantly different when females and males were compared. The 
ANCOVA results suggest that there was a significant difference between males and 
females in the treatment group compared to those in the control group: although both 
males and females benefited from DBS, females (in the treatment group) appear to have 
benefited more from DBS instruction than males (in the treatment group). This 
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statistically significant, interaction rejects the second null hypothesis (H = ^maies = 

/^ females J- 

Research question three: problem solving competency across race. 

This section presents the results of analyses intended to answer research question 
three: Are the observed effects of DBS different for students of different races? An 
analysis of covariance was used to assess whether there were significant differences 
between the average total problem solving scores of Black and Hispanic students, after 
controlling for problem solving pretest scores. The small number of Asian students (one 
each in control and treatment groups), White students (all three in control group) and 
Native American students (one in control group, two in treatment group), warranted their 
exclusion from the analyses. The following assumptions were tested a) independence of 
observations, b) normal distribution of the dependent variable, c) homogeneity of 
variance, d) linear relationships between the covariate and dependent variable, and e) 
homogeneity of regression slopes. All assumptions were met. The results indicate that the 
problem solving competencies of Black and Hispanic students in the treatment and 
control groups were not significantly different, F (1, 51) = 1.07, p = .305, eta = .02. The 
interaction of group, gender and race was however significant, F (1, 51) = 5.48, p < .05, 
eta = .097. Thus the combination of group, gender and race explains about ten percent of 
the difference in variance of problem solving competency between the control and 
treatment group. These results, while accepting the third null hypothesis suggests 
significantly different effects of DBS on Hispanic males and Black males. The means and 
standard deviations for Black and Hispanic females and males in both treatment and 
control groups are presented in Table XL 



Table XI 

Adjusted and Unadjusted Group, Race and Gender Means and Variability for Problem 
Solving Competency Using Problem Solving Pretest Scores as Covariate 

Unadjusted Adjusted 



N M SD M SE 

Control group 41 3214.07 2056.72 2831.31 285.93 

Black 21 2288.86 1118.61 2545.29 354.23 

Female 13 2264.31 1087.38 2441.11 426.42 

Male 8 2328.75 1243.06 2649.47 560.72 

Hispanic 20 4185.55 2376.71 3174.53 467.72 

Female 9 3774.44 2813.74 2579.79 719.53 

Male 11 4521.90 2030.37 4066.64 451.61 

Treatment 33 5139.88 1934.76 5305.72 313.18 

Black 19 5055.00 2019.20 5219.07 434.45 

Female 7 4999.00 2655.63 5431.57 636.90 

Male 12 5087.67 1679.83 5006.58 596.84 

Hispanic 14 5255.07 1882.59 5409.69 450.47 

Female 8 6371.63 1230.13 6174.24 620.48 
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Table XI continued 



Male 



3766.33 



1562.66 



4262.86 643.11 



Table XII 

Analysis of Covariance for Problem Solving Competency as a Function of Group, Race 
and gender Using Problem Solving Pretest Scores as Covariate 



Source 



df 



MS 



eta 



Problem solving pretest 



Race 



Group *Race 



Group*Gender*Race 



Error 



47604131.04 21.90 < .001 .30 



1276836.92 



.59 



2335186.45 1.07 



.447 .11 



.305 .02 



11919431.53 5.48 < .001 .097 



2202501.27 



The significant effects and interactions between group, gender and race are 
depicted in Figure XIII - XVI below. Figure XIII shows that females benefited from 
DBS, as their mean problem solving scores improved, however female Hispanic students 
saw a greater improvement in their mean problem solving competency score. 
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Estimated Marginal Means of Total Prob. Solving Score (Posttest) 
at Gender = (Females) 

Group 

— 

— 1 




- Control group 

1 - Treatment group 



Race (1 -Black 2 -Hispanic) 

Covariates appearing in the model are evaluated at the following values: Total Prob. Solving score (Pretest) 

= 2329.23 

Figure XIII. Comparison of Female Students' Problem Solving Scores in Treatment and 
Control Groups by Race 

Estimated Marginal Means of Total Prob. Solving Score (Posttest) 
at Gender = 1 (Males) 

Group 
— 

- Control group 

1 - Treatment group 




Race 



(1 - Black 2 - Hispanic) 



Covariates appearing in the model are evaluated at the following values: Total Prob. Solving score (Pretest! 

= 2329.23 



Figure XIV. Comparison of Male Students' Problem Solving Scores in Treatment and 
Control Groups by Race 
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Figure XIV, on the other hand shows Black males benefiting more from DBS than their 
Hispanic counterparts in problem solving competency. Within each race, the effects of 
DBS on female and male students can also be compared. Figure XV shows that among 



Estimated Marginal Means of Total Prob. Solving Score (Posttest) 
at Race = 1 (Black students) 

Gender 

— o - Female 
— 1 1 - Male 



„ 5000- 

OJ 

S 



« 4000- 



Ul 3000" 
UJ 




Group (0 - Control group 1 - Treatment group) 



Covariates appearing in the model are evaluated at the following values: Total Prob. Solving score (Pretest) 

= 2329.23 



Figure XV. Comparison of Problem Solving Competency Scores of Black Female and 
Male Students 



the Black students females had a slight edge over males although the mean problem 
solving competency scores of both groups improved after the treatment. Also, among the 
Hispanic students, Figure XVI below shows that females appear to have benefited more 
from DBS instruction than their male counterparts, reversing the edge males had prior to 
the problem solving pretest. The mean problem solving competency scores for Hispanic 
males in the treatment and control group are similar, implying that in general Hispanic 
males appear not to have benefited from the DBS instruction. 
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Estimated Marginal Means of Total Prob. Solving Score (Posttest) 
at Race = 2 (Hispanic students) 

Gender 

—o 0- Female 
~ 1 1-Male 




Group (0 - Control group 1 - Treatment group) 

Covariates appearing in the model are evaluated at the following values: Total Prob. Solving score (Pretest) 

= 2329.23 



Figure XVI. Comparison of Problem Solving Competency Scores of Female and Male 
Hispanic Students 

Research question four: problem solving competency across SES. 

This section presents results of the analysis intended to contribute to the 
resolution of research question four: are the problem solving competency differences 
(observed between treatment and control groups) consistent with SES? An analysis of 
covariance was conducted to ascertain whether there were differences in the effect of 
DBS on the problem solving competencies of students with different SES after 
controlling for problem solving pretest scores. The following assumptions were met: a) 
independence of observations, b) normal distribution of the dependent variable, c) 
homogeneity of variance, d) linear relationships between the covariate and dependent 
variable, and e) homogeneity of regression slopes. To enable easy comparison, SES 
values were grouped into three categories with equal widths: low (8 - 27.99), mid (28 - 
47.99) and high (48 - 66). The results indicated that the observed differences in problem 
solving competencies between students with different SES in the treatment and control 
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group were not significantly, F (2, 51) = .86, p = .428, eta = .03. No interactions with 
SES (e.g. group and gender; group, gender and race) were statistically significant. The 
means and standard deviations for the treatment and control groups are presented in 
Table XIII for low, mid and high SES groups. 

Table XIII 

Adjusted and Unadjusted SES Group Means and Variability for Problem Solving 
Competency Using Problem Solving Pretest Scores as Covariate 

Unadjusted Adjusted 



N M SD M SE 

Control Low SES 19 4133.11 2477.92 3485.69 379.48 

Mid SES 16 2392.63 1122.14 2193.09 471.35 

High SES 6 2494.33 1403.01 2809.76 666.68 

Treatment Low SES 18 5204.11 1848.27 5054.98 377.14 

Mid SES 12 4723.92 2108.43 4824.11 422.17 

High SES 3 6418.33 1696.65 6282.18 851.81 
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Table XIV 

Analysis of Covariance for Problem Solving Competency as a Function ofSES Group, 
Using Problem Solving Pretest Scores as Covariate 



Source 



df 



MS 



eta 



Problem solving pretest 1 47604131.04 21.90 < .001 .35 



SES 



4326850.82 



.08 



.147 .07 



Group*SES Group 



1875317.77 



.86 



.428 .03 



Error 



51 



2174182.62 



Differences in the problem solving competencies of the SES groups in the treatment and 
control groups were not significant. The slopes of graphs in Figure XVIII however show 



Estimated Marginal Means of Total Prob. Solving Score (Posttest) 




F 
group J _ Low SES 

—I 2 - Mid SES 
3 - High SES 



Group (o - Control group 1 - Treatment group) 

Covariates appearing in the model are evaluated at the following values: Total Prob. Solving score (Pretest) 

= 23B6.9B 

Figure XVII. Graph of Problem Solving Posttest Scores for Treatment and Control SES 
Groups 
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that high and medium SES groups appear to have benefited more than low SES groups 

from DBS, with higher SES students scoring the highest on average. 

Summary 

In the above sections analyses of covariance were conducted to compare the mean 
problem solving competency scores of a) control and treatment groups, b) female and 
male students, c) Black and Hispanic students (within and between the two groups with 
aggregation across gender), and d) low-SES, mid-SES and high-SES students. These 
analyses were intended to answer research questions one to four respectively: 1) Does 
DBS have any effect on the problem solving competencies of students in a high school 
traditional chemistry class? 2) Does the effect of DBS on problem solving competency 
vary depending on gender? 3) Does the effect of DBS on problem solving competency 
vary depending on race? 4) Does the effect of DBS on problem solving competency vary 
depending on SES? 

The above ANCOVA results suggest that a) DBS improves the problem solving 
competency of students in a high school traditional chemistry class, thereby disproving 
null hypothesis one, which states that there is no effect of DBS on problem solving 

competency (H = //control = //experimental), b) the effects of DBS on problem solving 

competency significantly varies depending on gender, rejecting null hypothesis two, 
which states that there is no difference in the effect of DBS depending on gender (H = 

^female = //male), c ) the effects of DBS on problem solving competency does not 

significantly vary depending on race, accepting null hypothesis three, which states that 

there is no difference in the effect of DBS depending on race (H G = //Black = //Hispanic), d) 



there is a statistically significant interaction between race and gender when the control 
and treatment groups were compared, suggesting that DBS instruction improves problem 
solving competency of Black females, Black males and Hispanic females but not 
Hispanic males, and e) the effects of DBS on problem solving competency does not 
significantly vary depending on SES, accepting null hypothesis four, which states that 

there is no difference in the effect of DBS depending on SES (H G = /^lowSes = /^midSES = 

/^HighSEs)- 

DBS and Chemistry Achievement 

This section presents analyses needed to answer research questions five to eight 
respectively: 5) Does DBS have any effect on chemistry achievement of students in a 
high school traditional chemistry class? 6) Does the effect of DBS on chemistry 
achievement vary depending on gender? 7) Does the effect of DBS on chemistry 
achievement vary depending on race? 8) Does the effect of DBS on chemistry 
achievement vary depending on SES? 

Research question five: DBS and chemistry achievement. 

The intent of research question five was to ascertain whether students who were 
given DBS instruction did better on a test of chemistry concepts than students in a control 
group. The results of the analysis are shared below. These results show that there was no 
significant difference in the chemistry achievement of students in the control and 
treatment groups. 

An ANCOVA was performed, with CCI pretest as covariate to investigate if DBS 
instruction produced any significantly different CCI scores for the treatment group 
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compared with the control group. The following assumptions were met: a) independence 
of observations, b) normal distribution of the dependent variable, c) homogeneity of 
variance, d) linear relationships between the covariate and dependent variable, and e) 
homogeneity of regression slopes. All assumptions were met. The results indicate that 
there were no significant differences in knowledge gained in chemical change and 
chemical energy concepts between students in the treatment and control groups, F (1, 51) 
= 0.45, p = .51, eta 2 = .04. The means and standard deviations for the two groups are 
presented in Table XV. Thus DBS instruction appears not to have any effect on the 
chemistry achievement of high school students in a traditional chemistry. 

Table XV 

Adjusted and Unadjusted Group Means and Variability for Chemistry Concepts 

Inventory (CCI) Using CCI Pretest Scores as Covariate 

Unadjusted Adjusted 



N M SD M SE 

Control group 41 13.27 2.40 13.00 0.35 

Treatment group 33 13.67 1.63 13.58 0.39 
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Table XVI 

Analysis of Covariance for CCI as a Function of Group, Using CCI Pretest Scores as 
Covariate 



Source of MS F p eta 2 



CCI pretest 1 7.63 2.33 .133 .44 

Group 1 1.47 .45 .506 .01 

Error 51 4.71 

Research question six: DBS and chemistry achievement across gender. 

The goal of research question six was to investigate whether any effects of DBS 
depended on gender. The results of the analysis are shared below. These results show that 
there was no significant difference in the chemistry achievement between female and 
male students in the control and treatment groups. 

An ANCOVA was then performed, with CCI pretest as covariate to investigate if 
DBS produced any significantly different CCI scores for female than male students. This 
analysis was intended to help resolve research question six. The following assumptions 
were tested and met: a) independence of observations, b) normal distribution of the 
dependent variable, c) homogeneity of variance, d) linear relationships between the 
covariate and dependent variable, and e) homogeneity of regression slopes. All 
assumptions were met. The results indicate that there were no significant differences in 
CCI scores when females and males from the treatment group were compared with those 
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in the control group, F (1, 51) = 0.91, p = .35, eta = .02. The means and standard 
deviations for the two groups are presented in Table XVII. 



Table XVII 

Adjusted and Unadjusted Gender Means and Variability for Chemistry Concepts 

Inventory (CCI) Using CCI Pretest Scores as Covariate 



Unadjusted Adjusted 



N M SD M SE 



Control group Female 22 12.73 2.64 12.78 0.52 

Male 19 13.89 1.97 13.26 0.46 

Treatment group Female 15 13.93 1.94 13.52 0.54 

Male 18 13.44 1.34 13.65 0.54 
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Table XVIII 

Analysis of Covariance for CCI as a Function of Gender, Using CCI Pretest Scores as 
Covariate 



Source of MS F p eta 2 



CCI pretest 1 7.63 2.33 .133 .044 

Group 1 1.47 .45 .506 .009 

Group*Gender 1 2.98 .91 .345 .017 

Error 51 3.28 

The differences between the CCI scores for female and male students in the 
treatment and control groups were not statistically significant. Figure XVIII, however 
shows that there was just a small gain in the CCI scores of both female and male students 
in the treatment group compared with those of the control group. However, females in the 
treatment group scored higher than their male counterparts. 
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Estimated Marginal Means of CCI Posttest(22) 

Gender 



E 13.00- 




(0- Female 1-Male) 



Group (0 - Control group 1- Treatment group) 

Covariates appearing in the model are evaluated at the following values: CCI Pretest (22) = 478 

Figure XVIII. Graph of CCI Posttest Scores for Female and Male Students in Treatment 
and Control Groups 



Research question seven: DBS and chemistry achievement across race. 

The intent of research question seven was to ascertain whether any effect of DBS 
instruction on chemistry achievement depended on race. The results of the analysis are 
shared below. These results show that there was no significant difference in the chemistry 
achievement of Black and Hispanic students in the treatment group when compared with 
those in the control group. 

An ANCOVA was then performed, with CCI pretest as covariate to investigate if 
DBS instruction produced any significantly different CCI scores for Black than Hispanic 
students. The following assumptions were met: a) independence of observations, b) 
normal distribution of the dependent variable, c) homogeneity of variance, d) linear 
relationships between the covariate and dependent variable, and e) homogeneity of 
regression slopes. All assumptions were met. The results indicate that there was no 
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significant differences in CCI scores when Black and Hispanic students from the 
treatment group are compared with those in the control group, F (1, 51) = 3.09, p = .085, 
eta = .06. The means and standard deviations for the two groups are presented in Table 
XIX. 



Table XIX 

Adjusted and Unadjusted Race Means and Variability for Chemistry Concepts Inventory 

(CCI) Using CCI Pretest Scores as Covariate 



Unadjusted Adjusted 



N M SD M SE 

Control group 41 13.27 2.40 13.00 .350 

Black 21 12.05 1.99 12.19 .430 

Female 13 11.46 1.85 11.80 .533 

Male 8 13.00 1.93 12.59 .675 

Hispanic 20 14.55 2.14 13.97 .574 

Female 9 14.56 2.60 13.77 .886 

Male 11 14.55 1.81 14.27 .566 

Treatment 33 13.67 1.63 13.58 .390 

Black 19 13.42 1.47 13.57 .540 
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Table XIX continued 

Female 7 13.86 1.35 13.83 .780 

Male 12 13.17 1.53 13.32 .740 

Hispanic 14 14.00 1.84 13.59 .557 

Female 8 14.00 2.45 13.22 .761 

Male 6 14.00 .63 14.14 .789 



Table XX 

Analysis of Covariance for CCI as a Function of Race, Using CCI Pretest Scores as 
Covariate 



Source of MS F p eta 2 



CCI pretest 1 7.63 2.30 < .001 .30 

Race 1 5.22 1.59 .447 .11 

Group*Race 1 10.14 3.09 .305 .02 

Group*Gender*Race 1 .28 .09 .771 .002 

Error 51 3.28 
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When females and males in the two races in the treatment group were compared to those 
in the control group no significant differences are found (i.e. the interaction, 
group *gender*race was not significant), F (1, 51) = .086, p = .771, eta = .002. 

Research question eight: DBS and chemistry achievement across SES. 

The intent of research question eight was to ascertain whether any effect of DBS 
instruction on chemistry achievement depended on SES. The results of the analysis are 
shared below. These results show that there was no significant difference in the chemistry 
achievement of students from low, mid and high SES subgroups when the treatment 
group was compared with the control group. 

An ANCOVA was performed, with CCI pretest as covariate to investigate if DBS 
produced any significantly different CCI scores for students in different SES groups. This 
was intended to assist in answering research question eight. The following assumptions 
were met: a) independence of observations, b) normal distribution of the dependent 
variable, c) homogeneity of variance, d) linear relationships between the covariate and 
dependent variable, and e) homogeneity of regression slopes. All assumptions were met. 
The results indicate that there are no significant differences in CCI scores when students 
from the treatment group are compared with those in the control group, by their SES 
groups F (2, 51) = .67, p = .52, eta 2 = .03. The means and standard deviations for the two 
groups are presented in Table XXI. 
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Table XXI 

Adjusted and Unadjusted SES group Means and Variability for Chemistry Concepts 

Inventory (CCI) Using CCI Pretest Scores as Covariate 



Unadjusted 



Adjusted 



N 



M SD 



M 



SE 



Control group 



41 



13.27 2.40 13.00 .350 



Low SES 



19 



13.95 2.55 13.32 .470 



Mid SES 



16 



12.69 2.21 



12.78 .577 



High SES 



12.67 2.16 12.87 .819 



Treatment group 



33 



13.67 1.63 13.58 .385 



Low SES 18 



13.78 1.40 14.03 .467 



Mid SES 12 



13.75 1.87 13.89 .544 



High SES 



12.67 2.31 12.57 1.047 
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Table XXII 

Analysis of Covariance for CCI as a Function ofSES Group, Using CCI Pretest Scores 
as Covariate 



Source of MS F p eta 2 



CCI pretest 1 7.63 2.33 .133 .04 

SES 1 2.49 .76 .472 .03 

Group*SES 2 2.19 .67 .518 .03 

Error 51 3.28 

Summary 

In the above sections analyses of covariance were conducted to compare the mean 
chemistry concepts inventory scores of a) control and treatment groups, b) female and 
male students, c) Black and Hispanic students, and d) low-SES, mid-SES and high-SES 
students. These analyses were intended to answer research questions five to eight 
respectively: 5) Does DBS have any effect on the chemistry achievement of students in a 
high school traditional chemistry class? 6) Does the effect of DBS on chemistry 
achievement vary depending on gender? 7) Does the effect of DBS on chemistry 
achievement vary depending on race? 8) Does the effect of DBS on chemistry 
achievement vary depending on SES? 

The above ANCOVA results suggest that a) DBS does not improve the chemistry 
achievement of students in a high school traditional chemistry class, thereby accepting 
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null hypothesis five, which states that there is no effect of DBS on problem solving 
competency (H = //control = //experimental), b) the effects of DBS on chemistry achievement 

does not vary depending on gender, accepting null hypothesis six, which states that there 
is no difference in the effect of DBS on chemistry achievement depending on gender (H 

= /^female = /^maie), c) the effects of DBS on chemistry achievement does not significantly 

vary depending on race, accepting null hypothesis seven, which states that there is no 

difference in the effect of DBS depending on race (H G = //Black = //Hispanic), d) the effects 

of DBS on chemistry achievement does not significantly vary depending on SES, 
accepting null hypothesis eight, which states that there is no difference in the effect of 

DBS depending on SES (H = // LowSES = // mid sES = //Hi g hSEs)- 

Correlation between Problem Solving Competency and Chemistry Concepts Score 

Research question nine: problem solving as predictor of chemistry scores. 

To investigate whether the level of chemistry concepts acquired by a student may 
be predicted by his/her problem solving competency, a regression analysis was 
performed. This analysis involved all students, without a differentiation of control and 
treatment groups. The following assumptions were tested and met, a) independence of 
observations, b) linearity, and c) the dependent variable (chemistry concepts score) was 
approximately normally distributed. Problem solving competency (M = 4320.13, SD = 
2331.54), significantly predicted the chemistry concepts score (M = 13.63, SD = 2.33), 
F (1, 80) = 31.03, p < .001, adjusted R 2 = .27. This implies that 27% of the variance in the 
chemistry concepts score is predicted by problem solving competency. Also an r value 
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(yR ) of 0.53 indicates a high effect size. The beta weight, presented in Table XXIII, 
indicate that when the problem solving competency increases by one unit, chemistry 
concepts score increase by 0.001 units. The preceding analysis therefore disproves the 
null hypothesis, which states that problem solving competency is not predictive of 

chemistry concepts score (U =jUcci = //problem solving). 

Table XXIII 

Simple Linear Regression Analysis for Problem Solving Competency Predicting 

Chemistry Concepts Score (N = 82) 

Variable B SEB p 

Problem Solving Competency 0.001 0.00 .53*** 

Constant 11.35 .47 

Note. R 2 = .27; F (1, 80) = 31.03, p < .001. 
***/?< .001. 

Summary 

This study investigated the effects of DBS on the problem solving competencies 
of high school students in a traditional chemistry class. It explored differences in the 
effects of DBS among groups by race, gender and SES. The study also ascertained if 
problem solving competency was predictive of chemistry achievement. ANCOVA was 
used to analyze the data. The findings are as follow: a) DBS significantly improved the 
problem solving competency of students in the study, b) DBS significantly improves the 
problem solving competency of both males and females, with a slight urge among 
females, c) the differences in the effects of DBS in improving problem solving 
competency among Black and Hispanic students in this study was not statistically 
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significant, however, Black students and Hispanic female students showed significant 
improvement in problem solving competency after the DBS instruction, d) DBS did not 
statistically significantly improve the problem solving competency of students of 
particularly SES group(s), and e) Problem solving competency is a strong predictor of 
higher chemistry concepts score among students in both treatment and control groups. 
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CHAPTER V 
CONCLUSIONS 
Introduction 

Science (and math) achievement in the United States has been on the decline over 
the past couple of decades, both nationally and internationally (Ornstein, 2010; U. S. 
Department of Education 2004, 2006; NEAP, 2005). The following achievement gaps 
have been identified as contributing to the decline: a) achievement gaps between males 
and females (NSF, 2008; Ornstein, 2010) and b) achievement gaps between White and 
Asian students on the one hand and Black and Hispanic students on the other (Ornstein, 
2010; Clewell & Ginorio, 1996; Creswell & Houston, 1980). Instructional methods such 
as DBS have the potential of improving students' creativity/critical thinking, problem- 
solving ability, science-process skills and consequently science achievement (Mehalik, 
Doppelt and Schunn 2008; Fortus, Dershimer, Krajcik, Marx and Mamlok-Naaman, 
2004, 2005; OECD, 2003; Chang, 2001a, 2001b). The purpose of this study was to 
investigate the effects of DBS on problem solving competency and science achievement, 
thereby contributing to the search for ways to close the achievement gap between males 
and females, students from different races and the rich and poor. Specifically, the 
questions addressed were: 

1) Does DBS have any effect on the problem solving competencies of students in 
a high school traditional chemistry class? 

2) Does the effect of DBS on problem solving competency depend on gender? 

3) Does the effect of DBS on problem solving competency depend on race? 

4) Does the effect of DBS on problem solving competency depend on SES? 
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5) Does DBS have any effect on the chemistry achievement of students in a high 
school traditional chemistry class? 

6) Does the effect of DBS on chemistry achievement vary depending on gender? 

7) Does the effect of DBS on chemistry achievement vary depending on race? 

8) Does the effect of DBS on chemistry achievement vary depending on SES? 

9) Is the problem solving competency of students in a traditional chemistry class 
predictive of their chemistry achievement? 

The results of the study suggest that DBS contributes to the problem solving 
competency of students, when treatment and control groups were compared. The 
observed effects of DBS on problem solving competency were also significantly different 
for students of different gender: females benefiting significantly more than males. The 
effect of DBS on the problem solving competency of Black and Hispanic students was 
not significantly different. However, Hispanic males in the treatment group did not score 
significant higher than those in the control group, suggesting that DBS instruction may 
not be useful in improving problem solving skills of students in this group. One reason 
that may account for this similarity between Hispanic males in the control and treatment 
groups is the observation that during the deign project a couple of groups that had more 
Hispanic males consistently sought to solve problems that were not only overly simplistic 
but also did not meet all the constraints laid out for the project. Most female students on 
the other hand were more focused and more resolved to meet the constraints of the 
project. Even though some did not produce a finished elaborate design, their 
determination was an indication of their acceptance of the challenge and consequently 
may have passed through the DBS instruction, acquiring intended skills. 
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The observed effects of DBS on problem solving competency scores were also 
found not to depend on SES. Also, DBS had no statistically significant effect on the 
chemistry achievement scores of students in the study. The results however suggest a 
strong correlation between problem solving competency and chemistry achievement. 
These findings will be discussed below followed by study limitations and implications for 
research and practice. 
Findings and Interpretation of Results 
Effects of DBS on Problem Solving Competency 

Research questions one to four were intended on assessing whether DBS had any 
effects on problem solving competency and if there was, how the effect differed 
depending on gender, race and SES. The results suggest a significant difference in 
problem solving competency between the treatment group and the control group. Due to 
the difficulty in measuring problem solving as a whole, the PISA problem solving 
protocol focuses on three areas of problem solving, namely, decision making, system 
analysis and design, and troubleshooting. These three components were also present in 
the DBS instruction particularly during the design project. 

First students needed to decide on exactly what problem they wanted to solve. 
This stage was not that challenging since the decision was driven primarily by interest. 
Decision making was however present in the other stages as well. The next stage of 
system design involved identifying the various components of the system. Students, in 
applying the design process were required to produce drawings showing different angles 
of their system, including a plan (top view) and an elevation (side view). During this 
stage they had to decide on the types of materials to use to produce the best desired 
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results. Students also had to determine the arrangement of system components that would 
produce the best results. Students had to make all these decisions cognizant of the 
chemistry concepts or reaction(s), which form the fundamental aspect of the system. This 
stage was quite challenging for the students, as they were clearly not used to this level of 
critical thinking. Students needed a lot of guidance and direction or else they were 
inclined to give up. In fact they preferred being told what to do at this stage. During the 
third stage when they had completed their design and the system was either not working 
at all or not working as anticipated, students had to troubleshoot by investigating possible 
causes. Even if the system initially worked students had to determine conditions for 
optimum results. During this and the previous stages students had to think critically in 
order to move on with their projects. The level of difficulty of each of these stages may 
have produced different learning environments for the students that may reflect in their 
performance on the problem solving assessment. 

The results of the problem solving competency pretest put the treatment and 
control groups at practically the same level in all three measured categories of problem 
solving. The use of ANCOVA as analysis further leveled the playing field by way of 
controlling for these pretest scores. The results showed that the treatment group 
performed better on the posttest than the control group on all three problem solving 
categories (Tables IV, VI and VIII). 

The above results support findings by Kolodner, Camp, Crismond, Fasse, Gray, 
Holbrool, Puntambekar, and Ryan, (2003) and Silk, Schunn and Strand, (2007). Their 
findings suggest that engaging students in design-based learning or problem-based 
learning within a science classroom has the potential of helping students develop problem 
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solving skills and scientific inquiry skills. The difference between the above referenced 

statements and the findings of the current study is that whereas the current study provides 

empirical evidence relating to a chemistry classroom the other is not based on any 

evidence. 

Effects of DBS on Problem Solving Competency across Gender 

Differences in science achievement between males and females have been 
adequately researched and established. Males appear to perform better than females in 
science particularly at the high school level (Ingels & Dalton, 2008; Bacharach, 
Baumeister & Furr, 2003; Jones, Mullis, Raizen, Weiss, &Weston, 1992). The search for 
ways to close this achievement gap is ongoing. In this regard the results of this study may 
contribute an understanding of instructional methods that support female excellence in 
science. 

The data from the current study suggests that there is no statistically significant 
difference the problem solving scores when males and females in the treatment are 
compared with those in the control groups (Table X). However, using the adjusted means 
of problem solving score in Table X for treatment and control groups, females (mean 
gain = 2564.19) appear to have benefited more from DBS instruction than males (mean 
gain = 1314.26). This relationship is displayed in Figure XL Although this interaction 
between group and gender is not statistically significant, it is worth replicating this study 
with a larger group, since a larger sample size may improve the reliability and power of 
analysis. Thus when females in the treatment group are compared with those in the 
control group, it becomes evident that females may stand to benefit more than males. 
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The items on the problem solving protocol were generally not gender biased and 
as a result they add to the credibility of any conclusions drawn from the above results. It 
is however, worth noting that if females would break through the glass ceiling in STEM 
related careers they must be continually supported in working past societal stereotypes 
about female and male roles: that males build while females nurture. 
Effects of DBS on Problem Solving Competency across Race 

Conversations involving achievement gaps cannot be complete without the 
inclusion of Native American, Black and Hispanic students' performance - trailing White 
and Asian Students. This study thus also sought to investigate the effects DBS may have 
on problem solving competency depending on race. Research question three sought to 
ascertain the effects of DBS instruction on the problem solving competency of different 
races so as to determine if DBS has a potential to help close the achievement gap 
associated with race. Due to circumstances beyond researcher control, the number of 
Black and Hispanic students in the study sample was larger than other races. Also, the 
numbers of Asian, Native American and White students in the study sample were too 
small to be included in the analysis. If they had been included deductions on the effects 
of DBS on problem solving competency for these groups would not be meaningful. 

The results (Table XII) show that there is no significant difference in the problem 
solving scores of Black and Hispanic students. Due to the fact that these results are close 
to being significant, the interactions are worth discussing to bring to the fore the potential 
differences between subgroups in terms of benefits of DBS. Figure XII depicts the 
similarity in the gains in problem solving score for Black and Hispanic females. Figure 
XIII shows the problem solving competency scores of Black males in the treatment group 
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increasing while those of their Hispanic counterparts remaining almost unchanged. The 
Hispanic males therefore appeared to have benefited from DBS: whereas the gains in 
problem solving scores for Black females and males were close (Figure XII), those of 
Hispanic females and males showed that the females scored higher (Figure XV). 

Historically, in the school where the study was conducted there has been a 
disproportionately high number of Black and Hispanic students in traditional science 
courses. The racial composition of the study sample vis-a-vis that is the school as a whole 
attests to this observation: 2.4% Asian, 48.8% Black, 41.5% Hispanic, 3.6% Native 
American and 3.6% White. The current distribution of students of different races in this 
school includes: 4.3% Asian, 41.7% Black, 23.3% Hispanic, 0.9% Native American and 
29.3% White. Most of the Black and Hispanic students in the classes studied give a 
couple of reasons for avoiding higher level courses. These include their a) lack of 
motivation and b) perception that science is difficult. Such students hitherto wish to get 
by with a grade of D, even in traditional courses. If DBS can improve the problem 
solving competency of these students it can go a long way in improving their self- 
confidence. Expanding this study to improve the power and reliability of the analysis is 
therefore important. Expansion may include involving students from more school sites. 
Effects of DBS on Problem Solving Competency across SES 

The results of the PISA 2003 problem solving competency assessment suggest a 
strong correlation between problem solving competency and SES (OECD, 2003). It is 
common knowledge that a high proportion of Black, Hispanic and Native American 
students fall within low SES in the United States. The implication of the PISA 2003 
problem solving assessment results is that students from these races are more likely to 
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have low problem solving competencies. The goal of research question four was thus to 
establish the correlation between SES and problem solving and to ascertain if DBS has a 
potential of improving the problem solving competency of low SES students. In their 
study of the effects of DBS on science achievement among middle school students by 
gender, socioeconomic status (SES) and race-ethnicity, Mehalik, Doppelt and Schuun 
(2008) reported that low-achieving African American students benefited the most from 
DBS. 

The results of the current study show that there is no significant difference 
between the problem solving skills of low SES, mid SES and high SES when students in 
the control and treatment groups are compared. It may however be noted that while the 
interaction between group and SES group was not significant (Table XIV), all three SES 
groups in the treatment groups had higher problem solving scores than their peers in the 
control group (Figure XVI). This suggests that all SES groups appear to have benefited 
from DBS in their problem solving competency. The insignificant effect of DBS on 
problem solving skills across SES group remained so even after reducing the SES groups 
to two (lowSES and midSES), negating any fear that that SES groupings earlier used may 
be unrepresentative of the larger population.. 
Effect of DBS on Chemistry Achievement across Gender, Race and SES 

The results of preliminary studies by Fortus, Dershimer, Krajcik, Marx and 
Mamlok-Naaman, (2005) and Puntambekar & Kolodner, 2005 imply that DBS and other 
inquiry-based pedagogies have the potential of helping students develop science 
knowledge. The results of the current study, however suggest that DBS does not have a 
statistically significant effect of chemistry achievement of the students in this study, 
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neither across gender, race nor SES. This implies that teachers and administrators are 
likely not to shun DBS or they may have a neutral attitude to using it in their 
classrooms/schools. However, if DBS improves problem solving competency, as 
suggested from the answer to research question one and problem solving competency 
correlated to chemistry achievement, then there will be a long-term benefit of DBS not 
only in improving problem solving competency but also indirectly improving chemistry 
achievement. 
Problem Solving Competency as Predictor of Chemistry Achievement 

Data from the PISA 2003 revealed a high correlation of 0.8, between problem 
solving competency and science achievement. The current study suggests that DBS has a 
significant effect on problem solving competency. If problem solving is a predictor of 
chemistry achievement then most of the students whose problem solving competency 
scores improved or students who scored high on the problem solving protocol would do 
well on the CCI. However, the results of the current study indicated no difference in the 
chemistry achievement between the treatment and control group, although there was a 
significant difference in problem solving score between the two groups. 

When all students in the study (treatment and control groups together) were 
pooled for the analysis there was a strong correlation between problem solving scores and 
CCI scores. In other words problem solving scores appeared to be a predictor of 
chemistry achievement. Hence the observation that even though problem solving 
competency on the treatment group improved but their chemistry achievement was not 
different from students in control group implies that the benefits of DBS towards 
chemistry achievement were not immediate. 
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During the design challenge tasks students were expected to use some of the 
concepts and knowledge tested by the Chemistry Concepts Inventory (CCI). Thus the 
design challenge tasks had direct bearing (knowledge-wise and concepts-wise) with what 
was assessed by the CCI. The PISA problem solving items however had no direct bearing 
on the design challenge. Despite these facts students in treatment group improved on 
problem solving items over the control group. This was not the case for Chemistry 
Concepts Inventory (CCI). A number of reasons could be advanced to account for the 
similar CCI scores for the treatment and control groups: a) Students in the treatment 
group may have been so focused on solving their specific problem that they lost attention 
of chemistry concepts that were tested, b) students in control group are more 
academically motivated and performed better, c) the DBS instruction did not foster the 
acquisition of broad-base chemistry knowledge: perhaps problem solving competency 
alone is not sufficient to enable students to score higher on the CCI. 
Problem- Solving Theories and Current Study 

Three problem- solving theories were used as lenses to understand the results of 
the current study, namely, constructivist, expert- novice and cognitive theories. Through 
the identification of need-based problems by students for resolution, to the construction 
of prototypes, various student actions may be interpreted by these theories. This section 
discusses if and how these theories were at play during the design challenge projects as 
students attempted solving specific problems. 

In attempting to solve their chosen problem, students tapped into their prior 
knowledge (misconceptions or otherwise). For instance students who were enrolled in the 
school's JROTC (Junior Reserve Officer Training Corps) program were more inclined to 
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identify MRE (meals ready to eat) as a source of chemical energy. MREs are food 
packages provided to military personnel. The package comes with a sachet of powdered 
magnesium, which upon mixing with water generated enough heat to warm up food. 
However, these students had no understanding of the chemical reaction responsible for 
the heat generated. Thus by the end of their projects such students exhibited better 
conceptual understanding of their domain. Also, at the beginning of the project students 
demonstrated little to no declarative conceptual understanding but by the end of the 
project could explain the origins of energy change. Throughout the project however, 
students had a hard time with deciding the procedure they needed to apply in solving the 
problem. More opportunities for students to attempt similar challenges may provide more 
insight to students thereby showing their growth in expert problem solving procedural 
skills. 

The five components of the DBS framework were intended to assist in the 
development of students' conceptual understanding of the problem they intended to 
solve. The framework also challenged students to elicit both declarative and automated 
skills in order to resolve problems. In other words beyond the problem being resolved in 
the current study, students would recognize the need to look inward to elicit knowledge 
and skills required to solve any problem on hand. Students in the current study were 
successfully guided to this realization since the teacher/researcher refrained from giving 
direct answers to student questions. At times some students were frustrated that they did 
not get direct answers. In the end however, they appreciated that they had the answer to 
most of their questions and needed to process deeper for themselves. 
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Constructivist, novice-expert and cognitive theories, as reviewed above explain 
the observed significant effect of DBS instruction in improving the problem solving 
competencies of students, when the control and treatment groups were compared. This 
relationship may be due to the fact the DBS framework and the five-step DBS learning 
cycle are consistent with these theories. By tapping into their prior knowledge students 
constructed new knowledge for themselves. Also, as they were challenged by the design 
problem they developed the declarative conceptual understanding and procedural 
knowledge needed for problem resolution. Finally, the desire by students to eliminate 
their cognitive dissonance was enhanced by their inspiration to resolve a problem in order 
to meet their own need(s). 
Summary 

The findings of the study may apply to high school students in a traditional 
chemistry course, in an urban district similar to that in this study. The findings are as 
follow: a) DBS significantly affects the problem solving competency of students, b) DBS 
improves the problem competency of both males and females, with a statistically 
insignificant urge among females, c) DBS does not produce a statistically significant 
difference in problem solving competency between Black and Hispanic students both 
whom appear to benefit from DBS except Hispanic males, d) DBS had no significant 
effect on problem solving competency depending on socioeconomic status, and e) 
Problem solving competency is a strong predictor of higher chemistry concepts score 
among both treatment and control groups. 
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Generalizations 

Due to the limitations of sampling and various defining sample characteristics, the 
generalizability of the findings of this study is restricted to: a) students of traditional 
chemistry who are taking chemistry for the first time, b) students who are generally not 
highly motivated to study chemistry, and c) schools in an urban school district with 
similar student characteristics as the one used in this study. This is because testing one 
school makes generalization difficult since the individual school tested may generate 
better or worse results for students using that particular educational instruction. The 
students in one school may be from a completely different socioeconomic background or 
culture and therefore cannot be a representative sample of the population. 
Delimitations of the Study 

This study limited itself to high school students in a traditional chemistry class. 
These students were sophomores who were taking chemistry for the first time and 
participated in the school district's pre-course assessment. The pre-course assessment 
results and CCI pretest scores were used to determine the treatment and control groups. 
For purposes of homogeneity of student chemistry learning experiences, students who 
had taken a semester one class in chemistry but who could not complete semester two 
during the year preceding this study were not included. 
Limitations of the Study 

There are five limitations to the study. First, the present study is not framed in a 
true experimental research design with random assignment of subjects to treatment and 
control groups. Therefore, the generalizability of the results to the total high school 
student population of the nation is limited. Like any other age group, high school students 
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are a very heterogeneous population. The fact remains that certain segments of the high 
school population, by virtue of their micro-culture, abilities, etc., may not be included. 
However, since the research design utilizes student-level data, the results can provide 
valuable preliminary information about the effects of DBS on student problem solving 
competency among different student groups. 

Secondly, the study does not necessarily establish cause-and-effect study 
relationships. This is because in such social science research there are a number of other 
variables that are either not present in the study context or are not apparent in the study. 

Third, anytime an instrument is used the results are subject to the known 
reliability and validity of that instrument. Although some information about the 
instrument in regard to reliability and validity is known, the instrument may have 
limitations in measuring what they purport to measure. Only subsequent research with 
other audiences and with other instruments will help further our understanding of the 
concepts being measured in the study. 

The fourth limitation of this study is the teacher as researcher. Due to teacher's 
familiarity with students, the teacher's observations may be biased or influenced. To 
minimize this peer observers were requested and visited two class periods as well as 
reviewed student work graded by teacher/researcher. 

Finally, the proportion of Asian and White students in the study sample was much 
lower than what pertains in the school as these students tend to participate in higher level 
chemistry classes. Hence, the few Asian and White students in the study were not 
included in the analyses that involved race. 
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Implications for Research 

Due to the exploratory nature of this study, the primary suggestion for future 
research is to build on the present study by replicating it in the future, with some 
modifications. These modifications include: a) the use of random sampling techniques 
that select a number of schools within one or more cities to increase the generalizability, 
b) to include various academic levels such Advance Placement and International 
Baccalaureate, c) include sufficient students from all races, d) include other science 
subjects such as physics and biology, and e) use DBS curriculum that involve more than 
one unit over a longer period than used in this study. 

Secondly, studies are clearly needed to develop a more broad-spectrum problem 
solving competency protocol. The PISA 2003 problem solving protocol measures only 
three aspects of problem solving: decision making, system analysis and design, and 
Trouble shooting. A more comprehensive protocol must perceive problem solving 
primarily as an internal and sequential process that includes cognitive, affective, and 
psychomotor behaviors. It must be stated that problem solving is a complex concept for 
measurement and includes ill-defined and well-defined problems. The later appear to be 
easier to measure than the former. 

Finally, if the strong correlation between problem solving and chemistry 
achievement is further established, research is needed to identify effective school-wide 
best practices and culture around the use of real- world problem solving to spur science 
achievement. After all, problem solving does not involve a set of skill limited to 
successful academic life but rather the very existence of the human race. 
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Implications for Practice 

The "Next Generation Science Standards" ( http://www.nextgenscience.org/next- 
generation- science- standards ) is about to be released, after going through the final phase 
of public review. One of the cornerstones of the new standards is the integration of 
science with engineering design. In retrospect, a large scale empirical investigation of the 
kind in this study is long overdue to identify mainstream trends as well as the deviations 
from the norm: how do different high school student groups respond to engineering 
design (or design based science)? The current study thus provides some insights as to 
how teachers should view the influences of DBS on their students. 

The strong correlation between problem solving and chemistry (and hopefully 
science) achievement suggests that a greater emphasis needs to be placed on problem 
solving earlier in the child's education. DBS can be viewed as one of the ways to 
incorporate problem solving instruction into science classrooms since for the study 
subjects and conditions, it has been shown to have a significant effect on problem solving 
competency. The challenge is the lack of a comprehensive design based science 
curriculum. Also, science teachers will need intensive training in DBS to make them 
effective in integrating science with design. 

The implications of the continuously widening achievement gaps between 
different student groups make the findings of this study compelling. According to the 
McKinsey consulting firm (2009), the gap in science and math achievement between 
1983 and 1998, cost the U.S. a Gross Domestic Product (GDP) of approximately $2 
trillion higher. The achievement gap between Black and Hispanic students on the one 
hand and white and Asian students by 1998 cost the U.S. about $400 to $500 billion. The 
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results of this study suggest that the use of DBS instruction could improve the problem 
solving competencies of Black students (in particular) as well as Hispanic students. DBS 
instruction could also improve female achievement since females respond well to it. 

Collaboration is one of the twenty first century skills that will enable students to 
be successful at the workplace. DBS instruction also serves to improve collaboration 
among students as it involves students in group projects. Collaboration provides students 
the opportunity to learn from each other and develop communication skills. 

Finally, staying current and well informed about research exploring the effects of 
design based science instruction on problem solving competency and academic 
achievement is an important exercise in science teacher professional development. 
Furthermore, science teachers should seek opportunities to share the implications of such 
research with school administrators, faculty members, and parents of children enrolled in 
their schools. 
Summary 

This study investigated the effects of DBS on the problem solving competencies 
of high school students in a traditional chemistry class. It explored differences in the 
effects of DBS among groups by race, gender and SES. The study also ascertained if 
problem solving competency was predictive of chemistry achievement. The findings are 
as follow: a) DBS significantly affects the problem solving competency of students, b) 
DBS improves the problem competency of both males and females, with a statistically 
insignificant urge among females, c) DBS does not produce a statistically significant 
difference in problem solving competency between Black and Hispanic students both 
whom appear to benefit from DBS except Hispanic males, d) DBS had no significant 
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effect on problem solving competency depending on socioeconomic status, and e) 
Problem solving competency is a strong predictor of higher chemistry concepts score 
among both treatment and control groups. 

Due the limitations of the study the following are recommendations for future 
research: a) the use of random sampling techniques that select a number of schools within 
one or more cities to increase the generalizability, b) to include various academic levels 
such Advance Placement and International Baccalaureate, c) include sufficient students 
from all races, d) include other science subjects such as physics and biology, and e) use 
DBS curriculum that involve more than one unit over a longer period than used in this 
study. 
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APPENDIX A 

The Barratt Simplified Measure of Social Status (BSMSS) 

Measuring SES 

Will Barratt, Ph.D. 

Circle the appropriate number for your Mother's , your Father's , your Spouse Partner's , and 
your level of school completed and occupation. If you grew up in a single parent home, circle 
only the score from your one parent. If you are neither married nor partnered circle only your 
score. If you are a full time student circle only the scores for your parents. 



Level of School Completed 


Mother 


Father 


Spouse 


You 


Less than 7 grade 


3 


3 


3 


3 


Junior high Middle school (9 grade) 


6 


6 


6 


6 


Partial lush school (10" 1 or 1 l m grade) 


9 


9 


9 


9 


High school graduate 


12 


12 


12 


12 


Partial collese (at least one year) 


15 


15 


15 


15 


College education 


18 


IS 


IS 


18 


Graduate degree 


21 


21 


21 


21 



Circle the appropriate number for your Mother's, your Father's . vour Spouse Partner's , and 
your occupation. If you grew up in a single parent home, use only the score from your parent. If 
you are not married or partnered circle only your score, If you are still a full-time student only 
circle the scores for your parents, If you are retired use your most recent occupation, 



Occupation 


Mother 


Father 


Spouse 


You 


Day laborer, janitor, house cleaner, farm worker, food 
counter sales, food preparation worker, busboy. 


5 


5 


5 


5 


Garbage collector, short-order cook, cab driver, shoe 
sales, assembly line workers, masons, basaage porter. 


10 


10 


10 


10 


Painter, skilled construction trade, sales clerk, truck 
driver, cook, sales counter or general office clerk, 


15 


15 


15 


15 


Automobile mechanic, typist, locksmith, fanner. 
carpenter, receptionist, construction laborer, hairdresser. 


20 


20 


20 


20 


Machinist, musician, bookkeeper, secretary, insurance 
sales, cabinet maker, personnel specialist, welder. 


25 


25 


25 


25 


Supervisor, librarian, aircraft mechanic, artist and 
artisan, electrician, administrator, military enlisted 
personnel, buyer. 


30 


30 


30 


30 


Nurse, skilled technician, medical technician, counselor. 
manager, police and fire personnel, financial manager. 
physical, occupational, speech therapist. 


35 


35 


35 


35 


Mechanical, nuclear, and electrical engineer, 
educational administrator, veterinarian, military officer, 
elementary, high school and special education teacher. 


40 


40 


40 


40 


Physician, attorney, professor, chemical and aerospace 
engineer, judge. CEO. senior manager, public official, 
psychologist, pharmacist accountant, 


45 


45 


45 


45 
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Level of School Completed Scoring 






1 


If you prew up with both parent* add Mother + Father and divide by 2. 
If you grew up with one parent enter that score to the right, 




2 


If you are married or partnered add Spouse + You and divide by 2, 

If you live alone enter Your score to the riaht. 
If you are a full-time student leave this blank. 






3 


Double your score from line 2. 

If you are a fall-time student leave this blank. 




4 


If you are a full-time student enter only your parents' score, 
Add line 1 and line 3 then divide by 3 (three) for a TOTAL EDUCATION 

Score should be between 3 and 2 1 





Occupation Scoring 



1 


If you grew up with both parents add Mother + Father and divide by 2. 
If you grew up with one parent enter that score to the right, 




2 


If you are married or partnered add Spouse + You and divide by 2, 
If you live alone enter Your score to the right. 
If you are a fall- time student leave this blank. 






3 


Double your score from line 2. 

If you are a fall-time student leave this blank. 




4 


If you are a full-time student enter only your parents' score, 
Add line 1 and line 3 then divide by 3 (three) for TOTAL OCCUPATION 

Score should be between 5 and 45 





TOTAL Score: 



Add TOTAL EDUCATION + TOTAL OCCUPATION: 

Score should be between S and 66 
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APPENDIX B PISA 2003 SAMPLE PROBLEMS AND SCORING RUBRIC 



Energy Needs 



Daily energy needs recommended for adults 







Men 


Women 




Age (years) 


Activity level 


Energy needed (kj) 


Energy necdec 


(tj) 


From 1 8 to 29 


Light 

Moderate 

Heavy 


10660 
11080 
14420 


8360 
8780 
9820 


From 30 to 59 


Light 

Moderate 

Heavy 


10450 
12120 
14210 


8570 
8990 
9790 


60 and above 


Light 

Moderate 

Heavy 


8780 

10240 

11910 


7500 
7940 
8780 



Activity level according to occupation 
Moderate: 



Light: 

Indoor sales person 
Office worker 
Housewife 



Teacher 

Outdoor salesperson 

Nurse 



Heavy: 

Construction worker 

Labourer 

Sportsperson 



ENERGY NEEDS - Question 1 

Mr Pavi'd Edison is 3 45-year-old teacher. What is bis recommended daily energy 

need in kj? 

Answer: kilo joules. 

Jane Gibbs is a 19-yegr old high jumper. One evening, some of Jane's friends 
invite her out for dinner at a restaurant. Here is the menu: 





MENU 


Jane's estimate of energy 
per serving (kj) 


Soups: 


Tomato Soup 


355 




Cream of Mushroom Soup 


585 


Main courses: 


Mexican Chicken 


960 




Caribbean Ginger Chicken 


795 




Pork and Sage Kebabs 


920 


Salads: 


Potato salad 


750 




Spinach, Apricot and Haxelnut Salad 


335 




Couscous Salad 


480 


Desserts: 


Apple and Raspberry Crumble 


1380 




Ginger Cheesecake 


1005 




Carrot Cake 


565 


Milkshakes: 


Chocolate 


1590 




Vanilla 


1470 



The restaurant also has a special fixed price menu. 



Fixed Price Menu 

30 zeds 

Tomato Soup 

Caribbean Ginger Chicken 

Carrot Cake 
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ENERGY NEEDS - Question 2 

Jgne keeps 3 record of whgt she egte egcb cjgy. Before 4 in net- on thgt dgy hertotgl 
intake of energy had. been 7520 kJ. 

Jane c|oes not want her total energy intake to go below or above her 
recommended daily amount by more than 500 kJ. 

Pecide whether the special "Fixed Price Menu" will allow Jane to stay within 
±500 kJ of her recommended, energy needs. Show your work. 



This problem is about finding a suitable time and date to go to the cinema. 

Isaac, a 15-year-old, wants to organise a cinema outing with two of his friends, 
who are of the same age, during the one -week school vacation. The vacation 
begins on Saturday, 24 th March and ends on Sunday, 1 st April, 

Isaac asks his friends for suitable dates and times for the outing. The following 
information is what he received, 

Ired; "I have to stay home on Monday and Wednesday afternoons for music 
practice between 2 : 30 and 3 : 30 " 

Stanley; "I have to visit my grandmother on Sundays, so it can't be Sundays, I 

havo -.;■!. ii Pf'Limin and drm't want to see it again." 

Isaac's parents insist that he only goes to movies suitable for his age and does not 
walk home, They will fetch the boys home at any time up to 10 p.m. 

Isaac checks the movie times for the vacation week. This is the information that 
he finds. 



T1VOU CINEMA 



Advance Booking Number: 01 924 423000 

24 hour phone number; 0192+ 420071 

Bargain Day Tuesdays; All films S3 

Films showing from Fri 23^ March for Iwo weeks: 



Children in the Net 

1 1 3 mins Suitable only for persons 



14:00 (Mon-Fri only) 
21:35 (Sat/ Sun only) 



of 1 2 years and over 



1'ok.amin 

105 mins 

13;40 (Daily) 
16:35 (Daily) 



Parental Guidance, General 

viewing, bul some scenes 
may be unsuitable for young 
children 



Monsters from the Deep 

164 mins 

, or rn;.,n, i , Suitable only for persons 

19:55 Fn/Sat only) P ,_ J , r 

oJ 1 B years and over 



Enigma 

144 mins 

1 5 ;00 (Mon-Fri only) 
18:00 (Sat/ Sun onlv) 



Suitable onlv for persons 



>f 12 



per 



vears and. over 



Carnivore 

148 mins 
18:30 (Daily) 



Suitable only for persons 
of 1 8 years and over 



King of the Wild 

1 17 mins 

14:35 (Mon-Fri only) 

1S;50 (Sat/ Sun only) 



Suitable for persons of 
all ages 



Cinema Outing 

CINEMA OUTING - Question 1 

Taking into account the information Isaac found on the movies, and the 
information he got from his friends, which of the six movies should Isaac and the 
boys consider watching! 1 

Circle "Yes" or "No" for each movie. 



Movie 


Should the three boys consider watching the movie? 


Children in the Net 


Yes /No 


Monsters from the Deep 


Yes /No 


Carnivore 


Yes / No 


Pokamin 


Yes /No 


Enigma 


Yes / No 


King of the Wild 


Yes /No 



CINEMA OUTING - Question 2 

iftbe three boys decided on going to "Children in the Net", which ofthe 
following dates is suitable for them? 

A. Monday, 26 th March 

B. Wednesday, 28* March 

C. Friday, 30 th March 
P. Saturday, 31 ' A March 
E. Sunday, 1 st April 
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Holiday 



This problem is about planning the best route for a holiday. 

Figures 1 and 2 show a map of the area and the distances between towns. 

Figure I . Map of roads between towns 



Kad 




Figure 2. Shortest road distance of towns from each other in kilometres. 



Anga/. 












Kado 


550 






Lapat 


500 


300 






\ ! ■, ■_' ii 1 


300 


850 


550 








Nuben 


500 




1000 


450 






Piras 


300 


850 


800 


600 


250 





Angaz 



Kado 



Lapat 



Megal 



Nube 



Piras 



HOLIDAY - Question I 

Calculate the shortest distance by ro^d between Nuben 3nd Kg do. 

Distance-. kilometres. 



HOLIDAY - Questran 2 



Zoe lives in Anggz. 5 be wgnts to visit K^dp gnd Upat. She cgn only travel up 
to 300 kilometres in gny one (fay. but cgn bregk her [ourney by camping 
overnight anywhere between towns. 
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Zoe will stay for two flights in eqch town, so thqt she qn spend, one whole t(gy 
sightseeing in eqch town. 

5how Zoe's itinerary by completing the following tqble to indicate where she 
stqys eqch night. 



Diiy 


Chcrmclit Stay 


1 


Camp-site between Angaz and Kado, 


2 




3 




4 




5 




6 




7 


Angaz 
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Transit System 



The following diagram shows part of the transport system of a city in 
Zedland, with three railway lines. It shows where you are at present, and 



where vou have to go. 

J O 



Line A 



Line C 



LincB 




Means a station on a 
railway line 

Means a station that is a 
junction where vou can change 
from one railway line to 
another (Lines A, B or C). 



The fare is hased on the number of stations travelled (not counting the 
station where vou start vour iournev). Each station travelled costs 1 zed. 

The time taken to travel between two adjacent stations is about 2 minutes. 

The time taken to change from one railway line to another at a junction is 

about 5 minutes. 
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TRANSIT SYSTEM - Question T 

The diagram indicates a station where you are currently at ("From here"), and 
the station where you want to qo ("To here"). Mark on the diagram the best 
route in terms of cost and time, and indicate below the fare you have to pay, and 
the approximate time for the journey. 

Fare: zeds. 

Approximate time for journey: minutes, 
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Library System 



The John Hobson High School library has a simple system for lending 


hooks: for staff members the loan period is 28 days and for students the 


loan period is 7 days. The following is a decision tree diagram showing this 


simple system: 


| STARTl 


A 


/ Is the borrower 


\ Yes 


Loan period is 

28 days 




n. a staff member? 


/ * 


^ 


No 




< 


' 






Loan period is 






7 days 









The Greenwood High School library has a similar, but more complicated, 
lending system: 

• All publications classified as "Reserved" have a loan period of 2 days. 

• For books (not including magazines) that are not on the reserved list, 
the loan period is 28 days for staff, and 14 days for students. 

• For magazines that are not on the reserved list, the loan period is 
7 days for everyone. 

• Persons with any overdue items are not allowed to borrow anything. 
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LIBRARY SYSTEM - Question 1 

You are a student at Greenwood High School, and you do not have any 
ovef4ue items from the library. You want to borrow a book that is not on the 
reserve^ list. How long can you borrow the book for? 

Answer: days. 

LIBRARY SYSTEM - Question 2 

Develop a decision tree diagram (or the Greenwood High STAR 

School Library system so that an automated checking system 

can be designed to deal with book and magazine loans at the 

library. Your checking system should be as efficient as possible 

(i.e. it should have the least number of checking steps). Note 

that each checking step should have only two outcomes and the 

outcomes should be labelled appropriately (e.g. "Yes" and "No"). 
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Design by Numbers 



Design by Numbers is a design tool for generating graphics on computers. 
Pictures can be generated by giving a set of commands to the program. 

Study carefully the following example commands and pictures before 
answering the questions. 



Paper 




Paper SO 




Paper 100 



Paper 


k 1 ! 


Paper 100 


Pen 100 


\H 


PenO 


Line 20 80 60 




Line 20 20 80 20 
Line 80 20 SO 80 
Line SO 80 20 20 
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DESIGN BY NUMBERS© - Question 1 

Which of the following commands generated the graphic shown be lows' 

A. Paper 

B. Paper 20 

C. Paper 50 
P. Paper 75 




DESIGN BY NUMBERS© - Question 2 

Which ofthe following set of commands generated the graphic shown below? 

A. Paper 100 Pen Line 80 20 80 60 

B. Paper Pen 100 Line 80 20 60 80 

C. Paper 100 PenO Line 20 80 80 60 
P. Paper Pen 100 Line 20 80 80 60 

DESIGN BY NUMBERS© - Question 3 

The following shows an example ofthe "Repeat" command- 

The command "Repeat A 50 80" telL 
P o I I I ~^ e program to repeat the actions in 

!^ cn 10 °, ™„„ \B I I brackets {}, for successive values of 

KepeatA iU aU . — SB 

{ HI | | A from A=50toA-80. 

Line 20 A +0 A 





Write commands to generate the following graphic: 
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Course Design 



COURSE DESIGN - Question 1 

Egch student will take four subjects per year, thus completing 12 subjects in three years. 

A student can only take a subject at a higher level if the student has completed the 
lower level(s) of the same subject in a previous year. For example, you can only 
take Business Studies Level 3 after completing Business Studies Levels 1 and 2. 

In addition, Electronics Level 1 can only betaken after completing Mechanics Level 1, 
and Electronics Level 2 can only betaken after completing Mechanics Level 2. 



3 


El 


Electronics Level 1 




4 


E2 


Electronics Level 2 




5 


Bl 


Business Studies Level 1 




6 


B2 


Business Studies Level 2 


7 


B3 


Business Studies Level 3 


8 


CI 


Computer Systems Level 1 




9 


CI 


Computer Systems Level 2 




10 


C3 


Computer Systems Level 3 




11 


Tl 


Technology and Information Management Level 1 




12 


T2 


Technology and Information Management Level 2 
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Pecicfe which subjects should, be offered for which yegr, by completing the 
following tgble. Write the subject codes in the tgble. 





Subject 1 


Subject 2 


Subject 3 


Subject 4 


Year 1 










Year 2 










Year i 











Children's Camp 



The Zedish Community Service is organising a five-day Children's Camp. 
46 children (26 girls and 20 boys) have signed up for the camp, and 8 adults 
(4 men and 4 women) have volunteered to attend and organise the camp. 



Table 1 . Adults 



Table 2. Dormitories 



Mrs Madison 


Mrs Carroll 


Ms 


Grace 


Ms 


Kelly 


Mr 


Stevens 


Mr 


Neill 


Mr 


Williams 


Mr 


Peters 



Name 


Number of beds 


Red 


12 


Blue 


8 


Green 


8 


Purple 


8 


(3 range 


8 


Yellow 


6 


White 


6 



Dormitory rides: 

1 . Boys and girls must sleep in separate dormitories. 

2. At least one adult must sleep in each dormitory. 

3. The adult(s) in a dormitory must he of the same 
gender as the children. 



160 



CHILDREN S CAMP - Question 1 

Dormitory Allocation 

Fill the table to allocate the 4-6 children and 8 adults to dormitories, keeping to 
gll the rules. 



Name 


Number of boys 


Number of girls 


Name(s) of adult(s) 


Red 








Blue 








Green 








Purple 








Orange 








Yellow 








White 
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Irrigation 

Below is a diagram of a system of irrigation channels for watering sections 
of crops. The gates A to H can be opened and closed to let the water go 
where it is needed. When a gate is closed no water can pass through it. 




^Out 



This is a problem about finding a gate which is stuck closed, preventing 
water from flowing through the system of channels. 

Michael notices that the water is not always going where it is supposed to. 

He thinks that one of the gates is stuck closed, so that when it is switched to 
open, it does not open. 



IRRIGATION - Question 1 

Michael uses the settings given in Table 1 to test the gates. 

Table I . Gate Settings 



A 


B 


C 


D 


E 


F 


G 


H 


< )pen 


Closed 


Open 


Open 


Closed 


Open 


Closed 


Open 



With the g^te settings as given in Table 1, on the diagram below c|raw all the 
possible paths for the flow of water. Assume that all gates are working according 
to the settings. 
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IRRIGATION - Question 2 



Michael finds that, when the gates have the Table 1 settings, no water flows 
through, Indicating that at least one ofthe gates set to open is stuck closed. 

Decide for each problem case below whether the water will flow through all the 
way. Circle "Yes" or "No" in each case. 



Problem Case 


Will water Jlow through all the way? 


Gate A is stuck closed. All other gates 
are working properly as set in Table 1 . 


Yes / No 


Gate D is stuck closed. All other gates 
arc working properly as set in Table 1 . 


Yes / No 


Gate F is stuck closed. All other gates 
are working properly as set in Table 1 . 


Yes / No 



IRRIGATION - Question 3 

Michael wants to be able to test whether gate D is stuck closed. 

In the following table, show settings for the gates to test whether gate D is stuck 
closed when it is set to open. 



Settings for gates |each one open or closed) 
A B C D E 



G 



H 
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Freezer 



Jane bought a new cabinet-type freezer. The manual gave the following 
instructions: 

• Connect the appliance to the power and switch the appliance on. 

• You will hear the motor running now. 

• A red warning light (LED) on the display will light up. 

• Turn the temperature control to the desired position. Position 2 is normal. 



Position 


Temperature 


1 


-15°C 


2 


-18°C 


'5 


-21°C 


4 


-25°C 


5 


-32°C 



• The red warning light will stav on until the freezer temperature is low 
enough. This will take 1 - 3 hours, depending on the temperature you set. 

• Load the freezer with food after four hours. 

Jane followed these instructions, but she set the temperature control to 
position 4. After four hours, she loaded the freezer with food. 

After eight hours, the red warning light was still on, although the motor was 
running and it felt cold in the freezer. 
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FREEZER - Question T 

Jane read "the manual again to see if she had done something wrong. She found 
the following six warnings: 

1. Do not connect the appliance to an unearthed power- point. 

2. Do not set the freezer temperatures lower than necessary (-18 °C is normal). 

3. The ventilation grills should not be obstructed- This could decrease the freezing 
capability of the appliance. 

4. Do not freeze lettuce, radishes, grapes, whole apples and pears, or fatty meat. 

5. Do not salt or season fresh food before freezing. 

6. Do not open the freezer door too often. 

Ignoring which of these six warnings could have caused the delay in the warning 
light going out? 

Circle "Yes" or "Ho" for each of the six warnings. 



Wr 



nrniin 



Could ignoring the warning have caused a delay in 
the warning light going out? 



Warning 1 



Yes / N< 



War 



n in 



cr 1 



Yes / No 



War nine 3 



Yes / N< 



Warning 4 



Yes / No 



Warning 5 



Yes / N< 



Warning 6 



Yes / N< 
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FREEZER - QUESTION 2 

Jane wonderecj whether the warning light was Functioning properly. Which of 
the following actions an4 observations woul4 suggest that the light was working 
properly? 

Circle "Yes" or "Ho" (or each of the three cases. 



Action and Observation 


Does the observation suggest that the 
warning light was working properly? 


She put the control to position 5 
and the red lijdit went off. 


Yes / No 


She put the control to position 1 
and the red light went off. 


Yes / No 


She put the control to position 1 
and the red light stayed on. 


Yes /No 
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Appendix C 

Features of the Three Types of Problem Solving skills 



Decision making 



System analysis 

and design 



Trouble shooline 



Coal CI 



loosing among 
alternatives under 
constraints 



Identifying the 
relationships 
between parts of 
a system and /or 
designing a system 
to express the 
relationships 
between parts 



Diagnosing and 
correcting a 
faulty or under- 
performing system 
or mechanism 



Processes 

involved 



Understanding a 

situation where 
there exist several 
alternatives and 
constraints and a 
specified task 



Understanding the 
information that 
characterises a 
given system and 
requirements 
associated with a 
specified task 



Understanding the 
main features of a 
system or 
mechanism and its 
malfunctioning, and 
the demands of a 
specific task 



Identifying relevant 
constraints 

Representing the 

possible 

alternatives 



Identifying relevant 
parts of the system 

Representing the 
relationships 
among parts of the 
system 



Identifying causally 
related variables 

Representing the 
functioning of the 
svstem 



Making a decision 

amongst 

alternatives 



Analysing or 
designing a system 
that captures the 
relationships 
between the parts 



Diagnosing the 
malfunctioning of 
the system and /or 
proposing a solution 



Checking and 
evaluating the 
decision 



Checking and 
evaluating the 
analysis or the 
design of the system 



Checking and 
evaluating the 
diagnosis and 
solution 



Communicating 
or justifying the 



C ommuni eating 
the analysis or 
justifying the 
proposed design 



Communicating or 
justifying the 
diagnosis and the 
solution 



Possible 

sources of 

complexity 



Number of 
constraints 



Number of inter- 
related variables 
and nature of 
relationships 



Number of inter- 
related parts to the 
svstem or mechanism 
and the ways in which 
these parts interact 



Number and type 
of representations 
used (verbal. 



Number and type 
of representations 
used (verbal. 



Number and type 
of representations 
used (verbal, 



pictorial, numerical) pictorial, numerical) pictorial, numerical) 
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Appendix 


D 










PISA 2003 Problem Solving Scale 










Level 












3 
















700 


» Transit System Gl 

i 


» Library System Q2 
















i 


• Child rens Camp QT 










600 \ 


1 
l Energy Weeds Q2 
I Holiday G2 f 


> Course Design G! 
Desion bv Numbers*; Q3 












2 

1 


t> Holiday Ol | Freezer QZ 
n Design by Numbers* Q2 o Freezer Q1 

l> Design by Numbers Q1 > ' "nation <« 
3 J n Irrigation Q3 
it Cinema Outing Q1 

SOO 








' ' II r lucjLtui I <i_j I 








( 


> Cinema Outing G2 












i 


i Library System Ql 








Below 
Level 1 


400 


























< 


> Energy Needs Q1 












decision 
making 


System analysis 

t j 

and design 


] rouble 
shooting 
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APPENDIX E 

ASSESSING ACADEMIC GAINS FROM THE HEATING/COOLING UNIT 

Chemical Concepts Inventory Sample Items: 

This inventory consists of 22 multiple choice questions. Carefully consider each question 
and indicate the one best answer for each. Several of the questions are paired. In these 
cases, the first question asks about a chemical or physical effect. The second question 
then asks for the reason for the observed effect. 

1. Which of the following must be the same before and after a chemical reaction? 

a. The sum of the masses of all substances involved. 

b. The number of molecules of all substances involved. 

c. The number of atoms of each type involved. 

d. Both (a) and (c) must be the same. 

e. (e) Each of the answers (a), (b), and (c) must be the same. 

2. Assume a beaker of pure water has been boiling for 30 minutes. What is in the bubbles 
in the boiling water? 

a. Air. 

b. Oxygen gas and hydrogen gas. 

c. Oxygen. 

d. Water vapor. 

e. Heat. 

3. A glass of cold milk sometimes forms a coat of water on the outside of the glass (Often 
referred to as 'sweat'). How does most of the water get there? 

a. Water evaporates from the milk and condenses on the outside of the glass. 

b. The glass acts like a semi-permeable membrane and allows the water to pass, but 
not the milk. 

c. Water vapor condenses from the air. 

d. The coldness causes oxygen and hydrogen from the air combine on the glass 
forming water. 

4. What is the mass of the solution when 1 pound of salt is dissolved in 20 pounds of 
water? 

a. 19 Pounds. 

b. 20 Pounds. 

c. Between 20 and 21 pounds. 

d. 21 pounds. 

e. More than 21 pounds. 
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5. The diagram represents a mixture of S atoms and O2 molecules in a closed container. 




C>2 molecule 



S atom 



Which diagram shows the results after the mixture reacts as completely as possible 
according to the equation: 



2S + 30 2 -> 2S0 3 



DUO Jo 
o 



m 




110 



m 



m 




(a) 



(b) 



(c) 



(<*) 



00 



6. The circle on the left shows a magnified view of a very small portion of liquid water in 
a closed container. 



Key 
£1 "Water 
O Oxygen f 
• Hydrogen 





Liquid "Water 



Evaporated "Water 



What would the magnified view show after the water evaporates? 




7. True or False? When a match burns, some matter is destroyed. 
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a. True 

b. False 

8. What is the reason for your answer to question 7? 

a. This chemical reaction destroys matter. 

b. Matter is consumed by the flame. 

c. The mass of ash is less than the match it came from. 

d. The atoms are not destroyed, they are only rearranged. 

e. The match weighs less after burning. 

9. Heat is given off when hydrogen burns in air according to the equation 

2H 2 + 2 -* 2H 2 
Which of the following is responsible for the heat? 

a. Breaking hydrogen bonds gives off energy. 

b. Breaking oxygen bonds gives off energy. 

c. Forming hydrogen-oxygen bonds gives off energy. 

d. Both (a) and (b) are responsible. 

e. (a), (b), and (c) are responsible. 

10. Two ice cubes are floating in water: 



Water 




After the ice melts, will the water level be: 

a. higher? 

b. lower? 

c. the same? 

11. What is the reason for your answer to question 10? 

a. The weight of water displaced is equal to the weight of the ice. 

b. Water is more dense in its solid form (ice). 

c. Water molecules displace more volume than ice molecules. 

d. The water from the ice melting changes the water level. 
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e. When ice melts, its molecules expand. 

12. A 1.0-gram sample of solid iodine is placed in a tube and the tube is sealed after all of 
the air is removed. The tube and the solid iodine together weigh 27.0 grams. 



■ Iodine solid 



V- 



The tube is then heated until all of the iodine evaporates and the tube is filled with iodine 
gas. Will the weight after heating be: 



a. 


less than 26.0 grams. 


b. 


26.0 grams. 


c. 


27.0 grams. 


d. 


28.0 grams. 


e. 


more than 28.0 grams. 



13. What is the reason for your answer to question 12? 

a. A gas weighs less than a solid. 

b. Mass is conserved. 

c. Iodine gas is less dense than solid iodine. 

d. Gasses rise. 

e. Iodine gas is lighter than air. 

14. What is the approximate number of carbon atoms it would take placed next to each 
other to make a line that would cross this dot: ■ 

a. 4 

b. 200 

c. 30,000,000 

d. 6.02 x 10 23 

15. Figure 1 represents a 1.0 L solution of sugar dissolved in water. The dots in the 
magnification circle represent the sugar molecules. In order to simplify the diagram, the 
water molecules have not been shown. 
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2.0 L 





Figure 1 



Figure 2 



Which response represents the view after 1.0 L of water was added (Figure 2). 








16. 100 mL of water at 25 °C and 100 mL of alcohol at 25 °C are both heated at the same 
rate under identical conditions. After 3 minutes the temperature of the alcohol is 50°C. 
Two minutes later the temperature of the water is 50°C. Which liquid received more heat 
as it warmed to 50°C? 

a. The water. 

b. The alcohol. 

c. Both received the same amount of heat. 

d. It is impossible to tell from the information given. 

17. What is the reason for your answer to question 16? 

a. Water has a higher boiling point then the alcohol. 

b. Water takes longer to change its temperature than the alcohol. 

c. Both increased their temperatures 25°C. 

d. Alcohol has a lower density and vapor pressure. 

e. Alcohol has a higher specific heat so it heats faster. 

18. Iron combines with oxygen and water from the air to form rust. If an iron nail were 
allowed to rust completely, one should find that the rust weighs: 

a. less than the nail it came from. 

b. the same as the nail it came from. 

c. more than the nail it came from. 

d. It is impossible to predict. 
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19. What is the reason for your answer to question 18? 

a. Rusting makes the nail lighter. 

b. Rust contains iron and oxygen. 

c. The nail flakes away. 

d. The iron from the nail is destroyed. 

e. The flaky rust weighs less than iron. 

20. Salt is added to water and the mixture is stirred until no more salt dissolves. The salt 
that does not dissolve is allowed to settle out. What happens to the concentration of salt 
in solution if water evaporates until the volume of the solution is half the original 
volume? (Assume temperature remains constant.) 



Solution- 



Halfofthe 




water evaporates 



Solid salt 




Solution 



Solid salt 



The concentration 

a. increases. 

b. decreases. 

c. stays the same. 

21. What is the reason for your answer to question 20? 

a. There is the same amount of salt in less water. 

b. More solid salt forms. 

c. Salt does not evaporate and is left in solution. 

d. There is less water. 

22. Following is a list of properties of a sample of solid sulfur: 

i. Brittle, crystalline solid. 

ii. Melting point of 1 13°C. 

iii. Density of 2. 1 g/cm . 

iv. Combines with oxygen to form sulfur dioxide 

Which, if any, of these properties would be the same for one single atom of sulfur 
obtained from the sample? 

a. i and ii only. 

b. iii and iv only. 
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c. iv only. 

d. All of these properties would be the same. 

e. None of these properties would be the same. 
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APPENDIX F 

CHEMISTRY CONCEPTS AND BIG IDEAS IN HEATING AND COOLING UNIT 



Subsystem & Big Idea 


Key concepts 


Reaction I 

Energy released or absorbed during chemical 

transformations is dependent on the shape and 

structure of the particles involved in the 

transformation. 


• Matter is made up of particles that have mass and occupy 
space. 

• Particles have a unique composition. The composition of 
particles determines their physical and chemical properties. 

• Particles interact with each other; this interaction may result in 
an increase or decrease in temperature. 

• Exothermic reactions are measured by an increase in the 
temperature of the system. Endothermic reactions are measured 
by a decrease in the temperature of the system. 

• The composition of particles determines how they interact 
with each other. 

• Interactions are the attraction between particles. Interactions 
between particles may result in transformations. 

• Transformations involve changes in attractions between 
particles. 

• Generally, as the size of the cation/anion increases the final 
temperature of the reaction involving the rearrangement of 
these ions will be lower. 

• Higher energy levels are related to the size of the cation/anion. 

• The size of the cation/anion is directly related to the distance 
to the 

nucleus and the attraction of the valence electrons of one 
nucleus to another nucleus. 


Reaction II 

Energy released or absorbed during chemical 
transformations is dependent on the mass and 
temperature change in the system. 


• Mass affects the amount of energy in the system. 

• An increase in mass results in more particle interactions, and 
consequently increases the energy of the system. 

• The mass of a reactant affects the change of temperature of 
the system. 

• All reactions have a specific maximum amount of energy. 

• Increases/decreases in mass are not directly proportional to 
increases/ decreases in temperature. 

• Changes in temperature are directly proportional to changes in 

energy. 


Container 

Energy transfers from particles with high kinetic 
energy to particles with lower kinetic energy 
through collisions. 


• The container is made up of particles that have unique 
composition that determines how they interact with the 
environment. 

• Conduction is the mechanism by which energy is transferred 
when two objects are in contact. 

• Thermal conductivity is the transfer of kinetic energy through 
conduction. 

• Thermal conductivity is a unique property of matter. 

• The atomic mass and structure of a substance affect its ability 
to transfer energy between adjoining atoms. 

• Substances that transfer heat energy quickly are called 
conductors. 

• Substances that transfer heat energy slowly are called 

insulators. 
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APPENDIX G 



STUDENT ACTIVITY FOR CONTROL AND TREATMENT GROUPS 



Topic 


Experimental Group Activity 


Control Group Activity 


Pretests 


■ Students provide biographical 
data (race gender, SES) 

■ Students take pretests (CCI) 
and PISA Problem Solving 


■ Students provide biographical 
data (race gender, SES) 

■ Students take pretests (CCI) 
and PISA Problem Solving 


Introduction 

Engineering 

Design 


■ Students assigned to watch 
videos: "Engineering Design 
Process" and "Bombing 
Hitler's Dams". 

■ Class discussion to relate the 
two videos in terms of the 
design process. 

■ Students completed worksheet 
enabling them to think about 
how the engineering design 
process was applied by 
investigators in "Bombing 
Hitler's Dam" video. 

■ End-of-unit design challenge 
project presented, with options 
clearly stated on handout 


■ Students watched the 
"Bombing Hitler's Dams" 
video and had to write an 
essay about the role of science 
in ending WWII. 

■ No viewing of "Engineering 
design process" video 

■ No end-of-unit design 
challenge project 


*Unit 
Activities 


Conservation of Mass 

Students were provided with materials 

and required to design an experiment to 

determine if mass is conserved at the end 

of a chemical reaction. 

Types of Chemical Reactions 

• Students learned about seven 
types of chemical reactions: 
Combustion, Synthesis, Single 
displacement, Double 
displacement, Decomposition, 
Neutralization and Redox. 
(Lecture, worksheets, 
homework, experiments test) 

• Students performed 
experiments involving 
endothermic and exothermic 
reactions. 

• Students designed a prototype 
gas-powered rocket propelled 
by igniting a mixture of 
hydrogen and oxygen gases in a 
plastic tube (rocket). 

• Students required to design an 
experiment to determine the 
best solvent for paints (from 
double displacement reactions). 

• Students practiced how to write 
different types of chemical 


Conservation of Mass 

Students were presented with directions 

as to how to carry out the same 

experiment. 

Types of Chemical Reactions 

• Students learned about seven 
types of chemical reactions: 
Combustion, Synthesis, Single 
displacement, Double 
displacement, Decomposition, 
Neutralization and Redox. 
(Lecture, worksheets, 
homework, experiments, test) 

• Students performed 
experiments involving 
endothermic and exothermic 
reactions. 

• No gas-powered experiments 

• Students provided instructions 
on how to determine the best 
of three solvents for paint. 

• Students practiced how to 
write different types of 
chemical equations. 

• Students prepare wet cells and 
investigate factors that 
determine optimum voltage 
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equations. 

• Students prepare wet cells and 
investigate factors that 
determine optimum voltage 

• Mini Project -design challenge: 
students required to convert 
their wet cell into a dry 
(portable) cell to charge their 
cell phones any time. 

• Students design experiments to 
investigate factors that affect 
speed of chemical reactions. 

• Students perform electroplating 
experiment and research the 
applications of electroplating 
and sacrificial anode 

• "Connecting with an idea" and 
Design Cycle activity 

Stoichiometry 

• Students practice balancing of 
chemical equations 

• Students practice solving 
stoichiometric problems. 


• No mini project 

• Students provided instructions 
on how to investigate the 
effects of temperature, 
concentration, surface area 
and specific substances 
(catalysts) on reaction rate. 

• Students perform 
electroplating experiment (No 
applications research) 

• Students practice balancing of 
chemical equations 

• Students practice solving 
stoichiometric problems. 

• Students practice balancing of 
chemical equations 

• Students practice solving 
stoichiometric problems. 


Unit Design 

Challenge 

Project 


• Students design solutions to 
problems: options include 

a) using heat energy change 
from chemical reactions to 
solve problems at a winter 
camp, summer camp, 
dating, sports, etc. 

b) design an age-appropriate 
toy that uses chemistry 
concepts learned in this 
unit 

• "Connecting with an idea" and 
Design Cycle activity 

• Designs to be presented to 
classmates as a board of a 
company looking for ideas to 
invest in. 


• No unit design Project 


Posttests 


Complete 

• CCI 

• PISA Problem Solving 
Competency test 

• Design challenge open-ended 
survey (Appendix H) 


Complete 

•CCI 

• PISA Problem Solving Competency 

test 



^Unit on atomic structure and chemical bonding was completed before above unit. 
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APPENDIX H 



SITE PRINCIPAL APPROVAL 



*3* 


DENVER 
PUBLIC 

SCHOOLS 



DHWMl J MArU ftf Opf*WV*wtif" 



Principal ConsenI Form 



Research Background (to be completed by researcher) 

Title of the Study Effects of Design-Based Chemistry Instruction on the Science Problem-solving Skills 
and Science Achievement among Different Groups of High-School Students 



Name of Researcher Cobina Adu I flflgflfl 



Phone (720) 690-327 1 



Street address: 3812 S. Ceylon Way City: Aurora State: _CQ Zip: 80013 

E-mail: clartson@yahoo.co.uk 

II. Description of Research Proposal 

A group of students will be exposed to design-based Chemistry instruction (units culminate in a 2D or 3D 
artifact). Pretests and posttests will be administered to measure problem solving competency and science 
achievement. The purpose of die research is to investigate the effects of design-based Chemistry instruction on 
problem solving competency and science achievement across gender, race/ethnicity and socio economic status. 
A control group will be used. However, if the treatment is found to produce significantly higher student 
achievement and better problem solving competency, the control group will also be given the treatment. The 
research questions of the study are: a) Is Design Based Instruction in high school chemistry associated with 
higher performance on measures of problem solving and chemistry concepts? b) Are the effects of design based 
instruction, if they exist, consistent for students of different gender, ethnicity, and SES? Chemistry content to be 
taught is the DPS approve curriculum for .Active Chemistry. 
(Researcher is to provide ihe principal with a copy of the executive summary and the lime requiremenl form) 

III. Agreement Up be aompleted by principal) 

'■ III IC'jt /^"L Of)r A/aCA. ' . principal of George Washington high school, understand 

• the study and what ii requires of [he staff, students, and/or parents in my school, 

• thai ihe privacy and confidentiality of any staffer student will be protected, 

• that I have the right to allow or reject this research study to lake place at my school. 

• that I have the right to terminate live research study at any lime. 

• that I have the right to review all consent forms and research documents at any time during (he study and up to three years 
after the completion of the study. 

IT1 grant permission 1o the researcher to conduct the above named research in rny school as described in the proposal. 
D 1 DO NOT gram permission to the researcher to conduct ihe above named research in my school as described in the 

proposal. 
OT I understan d that data should be released only by the departments that own ihem. My staff and I shall not release data to the 

researebar"^ilhoi?N ap proya1from the district's Research Review Board. 
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DISTRICT APPROVAL LETTER 



DENVER PUBLIC SCHOOLS 

900 Grant Street, Rra 610, Denver, CO 80203 




September 13, 2012 



Your research project, Effects of design-based chemistry instruction on the science problem-solving skills and 
science achievement among different groups of high school students, has been reviewed and approved by the 
Denver Public Schools IRB for implementation based upon the following conditions: 

1 . The voluntary nature of the study is made clear to all potential participants including pupils, teachers, and 
administrators. Final approval for the study is contingent on the principals, students, and parents ' agreement to 
participate . 

2. The researchers agree to maintain the anonymity of the research participants as outlined in your proposal. 

3. All rules in the district's research procedures are followed including maintaining the anonymity of the district, the 
schools, and the study participants. 

4. If your request involves the release of data you agree to limit the use of said data to the terms specified in your 
application. The data will not be released to any third party and you agree not to copy, reproduce, disseminate 
transmit, license, sublicense, assign, lease, or release the data to any other party. All data should be maintained in 
a secure fashion with access being restricted to the persons identified in the research application to prevent 
unauthorized use of the data. Following the use of the data for the prescribed reasons the data should be 
destroyed. 

5. This letter does not reflect a commitment on behalf of Denver Public Schools towards the requestor. At any point, 
the approval status involving the release of data or access to students/staff for research may be withdrawn. A 
violation of any of the conditions within this letter and/or deceptive practices by the researcher will lead to 
immediate termination of all research privelages. Furthermore, the release of future data and/or research 
privelages may be indefinitely terminated. 

6. A report of the findings is made available to the Department of Accountability, Research & Evaluation at the 
conclusion of the study. 

7. This letter is returned by mail or via FAX (720-423-3646) prior to initiating your study with the requestor 
acknowledging agreement with the terms described above by signature. 



Please contact Accountability, Research & Evaluation at 720-423-3736 if you have any questions. 
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Please return this letter with the following statement verified by signature: 

I, Cobina Adtt Lartson . agree to abide by the conditions described in this document and will carry out my 
research practices in accordance with those conditions. I assume complete responsibility for the described study and 
will work according to best-practices when working with Denver Public Schools data and/or conducting scientific 
inquiry within the Denver Public Schools district. 



u 




Signature of Requestor 
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APPENDIX J 

PARENT CONSENT LETTER 

Effects of Design-Based Chemistry Instruction on the Science Problem-solving Skills and Science Achievement among Different 
Groups of High-School Students COMTRB protocol number: 12-1150 PI: Cobina Lartson 09/28/2012 

University of Colorado 3 st2 s. ceyion way 

DOflVGr Aurora CO, 80013 

Parent Consent 

Dissertation Research Title: Effects of Design-Based Chemistry Instruction on the Science Problem-solving Skills 

and Science Achievement among Different Groups of High-School Students 

Principal Investigator: Cobina Adu Lartson., Sell, of Ed. & Human Development, Univ. of Colorado Denver. 

You are being asked to participate in a program where data will be collected for the purposes of research 

investigating the effect of a classroom instruction on student achievement and problem-solving competency. 

BenefltsAVhy is this study being done? 

This program is being done to help the researchers learn how to increase the number of African Americans and Latinos 

in science, technology, engineering and mathematics careers. Results from the program may improve how teachers 

enable African American and Latino children to take harder courses in science and mathematics in high school. You 

may benefit from this research because of the positive effect it may have on your future. If the treatment is found to 

significantly improve problem solving skills, the researcher will ensure that students m control group obtain the 

benefits of the treatment in the same or subsequent units. 

The learning objectives of the research are as follow: To 

• Increase students' Chemistry content knowledge. 

• Improve student problem-solving skills; decision-making, critical thinking and troubleshooting skills. 

• Increase participant interest in taking science courses and having a STEM career. 

• Encourage students to use science to meet needs in their communities. 

What happens if you join this study? In order to achieve the above goals, we will hold a ten-week energy change 
unit for high school students (ages 12-18) during which students design a product intended to solve a specific problem. 
Another group of students will study the same content without a design component. Before and after taking the unit, 
we will measure student achievement and problem solving competency by using pre and posttests. The hours you put 
in towards the smdy is the same as regular class hours. You will be eligible for a credit towards paying for your tie-dye classroom 
project. You will respond to socioeconomic status, aud demograpliical surveys. 

Is participation voluntary? At any time you may decide not to stay in the study. Your decision will not affect your 
relationship with the researcher. You will not be punished in any way if you decide to leave the study, however you 
must compete the science content for the study since it is the district approved curriculum that is being used. 
What are the possible discomforts or risks? There is minimal risk and very slight discomfort to children who join in 
this study. Discomforts you may experience while m this study include being asked to complete a test and survey and 

perform a class presentation. 

Will your personal information be released to anyone by the researcher? I I Yes I ^ I No 

Regulatory organizations that ensure researcher compliance with laws that protect human subjects may see all 
information related to this research. All the information about you, including pre and posttests will be kept in a safe 
place: infinite campus/password-protected classroom computer. In this way, the risk of others finding out about your 
test scores and survey results is reduced to the barest minimum. All data and information collected will be destroyed 
within two years of completing the study. 

Who is paying for this study? This research is being conducted as a requirement for a Ph.D. through the University of 
Colorado Denver as an extremely low-budget student research study and at no cost to you. 

Who do I call if I have questions? Parents, please be aware that under the Protection of Pupil Rights Act. you have 
the right to review a copy of the questions asked of or materials that will be used with your students. If you would like 
to do so, please call the Principal Investigator, Mr. Cobina A. Lartson (720.690.3271 or c-lartson@yahoo.co.uk) or 
study supervisor. Dr. Geeta Verma at Geeta.Vemiafoucdenver.edu . You may also call the Human Subject Research 
Committee (HSRC) at 303.724.1055 
Agreement to participate in this study 

I I I agree to my child's participate in this research, l | I do not agree to my child's participate in this research. 

Student name 

Parent Signature: Date: 
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APPENDIX K 

STUDENT CONSENT LETTER 

Effects of Design-Based Chemistry Instruction on the Science Problem-solving Skills and Science Achievement among Different 
Groups of High-School Students COMIRB protocol number: 12-1150 PI: Cobina Lartson 09/28/2012 

i ■ i ■. e f~\ \ i Cobiiia Adu Lartson 

University of Colorado jgu s . ceyion w ay 

DGriVGr Aurora. CO 80013 

Student Consent 

Dissertation Research Title: Effects of Design-Based Chemistry Instruction on the Science Problem-solving Skills 
and Science Achievement among Different Groups of High-School Students 

Principal Investigator: Cobina Adu Lartson.. Sch. of Ed. & Human Development, Univ. of Colorado Denver. 
You are being asked to participate in a program where data will be collected for the purposes of research 
investigating the effect of a classroom instruction on student achievement and problem-solving competency. 
Benefits/Why is this study being done? 

This program is being done to help the researchers learn how to increase the number of African Americans and Latinos 
in science, technology, engineering and mathematics careers. Results from the program may improve how teachers 
enable African American and Latino children to take harder courses in science and mathematics m high school. You 
may benefit from this research because of the positive effect it may have on your future. If the treatment is found to 
significantly improve problem solving skills, the researcher will ensure that students in control group obtain the 
benefits of the treatment in the same or subsequent units. 
The learning objectives of the research are as follow: To 

• Increase students' Chemistry content knowledge. 

• Improve student problem-solving skills; decision-making, critical thinking and troubleshooting skills. 

• Increase participant interest in taking science courses and having a STEM career. 

• Encourage students to use science to meet needs in their communities. 

What happens if you join this study? In order to achieve the above goals, we will hold a ten-week energy change 

unit for high school students during which students design a product intended to solve a specific problem. Another 

group of students will study the same content without a design component. Before and after taking the unit, we will 

measure student achievement and problem solving competency by using pre and posttests. The hours you put 

in towards the study is the same as regular class hours. You will be eligible for a credit towards paying for your tie-dye 

classroom project. You will respond to socioeconomic status, and demographical surveys. 

Is participation voluntary? At any time you may decide not to stay in the study. Your decision will not affect your 

relationship with the researcher. You will not be punished in any way if you decide to leave the study however you 

must compete the science content for the study smce it is the district approved curriculum that is being used. 

What are the possible discomforts or risks? There is minimal risk and very slight discomfort to children who join in 

this study. Discomforts you may experience while in this study include being asked to complete a test and survey and 

perform a class presentation. 

Will your personal information be released to anyone by the researcher? I I Yes I »j| No 

Regulatory organizations that ensure researcher compliance with laws that protect human subjects may see all 
information related to this research. All the information about you, including pre and posttests will be kept in a safe 
place: infinite campus/password-protected classroom computer. In this way, the risk of others finding out about your 
test scores and survey results is reduced to the barest minimum. All data and information collected will be destroyed 
within two years of completing the study. 

Who is paying for this study? This research is being conducted as a requirement for a Ph.D. through the University of 
Colorado Denver as an extremely low-budget smdent research study and at no cost to you. 

Who do I call if I have questions? Parents please be aware that under the Protection of Pupil Rights Act, you have the 
right to review a copy of the questions asked of or materials that will be used with your students. If you would like to 
do so, please call the Principal Investigator, Mr. Cobina A. Lartson (720.690.3271 or clartson(@yahoo.co.uk ) or study 
supervisor, Dr. Geeta Verma at Geeta.VerniafSucdenver.edu . You may also call the Human Subject Research 
Committee (HSRC) at 303.724.1055. 
Agreement to participate in this study 

| I I agree to in this research. | I I do not agree to participate in this research. 

Student name 

Signature: Date: 
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